You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by vito jeng <vi...@is-land.com.tw> on 2017/11/02 06:09:31 UTC
[DISCUSS]KIP-216: IQ should throw different exceptions for different errors
Hi,
I'd like to start discuss KIP-216:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors
Please have a look.
Thanks!
---
Vito
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Thank you for your attention to this KIP and assistance.
---
Vito
On Wed, Dec 6, 2017 at 2:35 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for the info. There is no hurry. Was just curious :)
>
> -Matthias
>
>
> On 12/5/17 7:34 PM, vito jeng wrote:
> > Matthias,
> >
> > Still in progress. I've been busy in recent weeks with my job.
> >
> > But I believe I will update this KIP within few days.
> >
> >
> >
> >
> > ---
> > Vito
> >
> > On Tue, Dec 5, 2017 at 8:05 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Vito,
> >>
> >> is there any update with regard to this KIP?
> >>
> >>
> >> -Matthias
> >>
> >> On 11/5/17 6:11 PM, vito jeng wrote:
> >>> Thanks, Guozhang and Matthias. Your comments very useful for me.
> >>>
> >>> I'll update KIP and keep going on.
> >>>
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> matthias@confluent.io>
> >>> wrote:
> >>>
> >>>> Thanks for the KIP Vito!
> >>>>
> >>>> I agree with what Guozhang said. The original idea of the Jira was, to
> >>>> give different exceptions for different "recovery" strategies to the
> >> user.
> >>>>
> >>>> For example, if a store is currently recreated, a user just need to
> wait
> >>>> and can query the store later. On the other hand, if a store go
> migrated
> >>>> to another instance, a user needs to rediscover the store instead of a
> >>>> "plain retry".
> >>>>
> >>>> Fatal errors might be a third category.
> >>>>
> >>>> Not sure if there is something else?
> >>>>
> >>>> Anyway, the KIP should contain a section that talks about this ideas
> and
> >>>> reasoning.
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>> Thanks for writing up the KIP.
> >>>>>
> >>>>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>>>> categories of errors we want to notify the users, if we only wants to
> >>>>> distinguish fatal v.s. retriable then probably we should rename the
> >>>>> proposed StateStoreMigratedException / StateStoreClosedException
> >> classes.
> >>>>> And then from there we should list what are the possible internal
> >>>>> exceptions ever thrown in those APIs in the call trace, and which
> >>>>> exceptions should be wrapped to what others, and which ones should be
> >>>>> handled without re-throwing, and which ones should not be wrapped at
> >> all
> >>>>> but directly thrown to user's face.
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
> >> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I'd like to start discuss KIP-216:
> >>>>>>
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>>>
> >>>>>> Please have a look.
> >>>>>> Thanks!
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks for the info. There is no hurry. Was just curious :)
-Matthias
On 12/5/17 7:34 PM, vito jeng wrote:
> Matthias,
>
> Still in progress. I've been busy in recent weeks with my job.
>
> But I believe I will update this KIP within few days.
>
>
>
>
> ---
> Vito
>
> On Tue, Dec 5, 2017 at 8:05 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Vito,
>>
>> is there any update with regard to this KIP?
>>
>>
>> -Matthias
>>
>> On 11/5/17 6:11 PM, vito jeng wrote:
>>> Thanks, Guozhang and Matthias. Your comments very useful for me.
>>>
>>> I'll update KIP and keep going on.
>>>
>>>
>>>
>>> ---
>>> Vito
>>>
>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Thanks for the KIP Vito!
>>>>
>>>> I agree with what Guozhang said. The original idea of the Jira was, to
>>>> give different exceptions for different "recovery" strategies to the
>> user.
>>>>
>>>> For example, if a store is currently recreated, a user just need to wait
>>>> and can query the store later. On the other hand, if a store go migrated
>>>> to another instance, a user needs to rediscover the store instead of a
>>>> "plain retry".
>>>>
>>>> Fatal errors might be a third category.
>>>>
>>>> Not sure if there is something else?
>>>>
>>>> Anyway, the KIP should contain a section that talks about this ideas and
>>>> reasoning.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>> Thanks for writing up the KIP.
>>>>>
>>>>> Vito, Matthias: one thing that I wanted to figure out first is what
>>>>> categories of errors we want to notify the users, if we only wants to
>>>>> distinguish fatal v.s. retriable then probably we should rename the
>>>>> proposed StateStoreMigratedException / StateStoreClosedException
>> classes.
>>>>> And then from there we should list what are the possible internal
>>>>> exceptions ever thrown in those APIs in the call trace, and which
>>>>> exceptions should be wrapped to what others, and which ones should be
>>>>> handled without re-throwing, and which ones should not be wrapped at
>> all
>>>>> but directly thrown to user's face.
>>>>>
>>>>> Guozhang
>>>>>
>>>>>
>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'd like to start discuss KIP-216:
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>>>>>>
>>>>>> Please have a look.
>>>>>> Thanks!
>>>>>>
>>>>>> ---
>>>>>> Vito
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
Still in progress. I've been busy in recent weeks with my job.
But I believe I will update this KIP within few days.
---
Vito
On Tue, Dec 5, 2017 at 8:05 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Vito,
>
> is there any update with regard to this KIP?
>
>
> -Matthias
>
> On 11/5/17 6:11 PM, vito jeng wrote:
> > Thanks, Guozhang and Matthias. Your comments very useful for me.
> >
> > I'll update KIP and keep going on.
> >
> >
> >
> > ---
> > Vito
> >
> > On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Thanks for the KIP Vito!
> >>
> >> I agree with what Guozhang said. The original idea of the Jira was, to
> >> give different exceptions for different "recovery" strategies to the
> user.
> >>
> >> For example, if a store is currently recreated, a user just need to wait
> >> and can query the store later. On the other hand, if a store go migrated
> >> to another instance, a user needs to rediscover the store instead of a
> >> "plain retry".
> >>
> >> Fatal errors might be a third category.
> >>
> >> Not sure if there is something else?
> >>
> >> Anyway, the KIP should contain a section that talks about this ideas and
> >> reasoning.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>> Thanks for writing up the KIP.
> >>>
> >>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>> categories of errors we want to notify the users, if we only wants to
> >>> distinguish fatal v.s. retriable then probably we should rename the
> >>> proposed StateStoreMigratedException / StateStoreClosedException
> classes.
> >>> And then from there we should list what are the possible internal
> >>> exceptions ever thrown in those APIs in the call trace, and which
> >>> exceptions should be wrapped to what others, and which ones should be
> >>> handled without re-throwing, and which ones should not be wrapped at
> all
> >>> but directly thrown to user's face.
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'd like to start discuss KIP-216:
> >>>>
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>
> >>>> Please have a look.
> >>>> Thanks!
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Vito,
is there any update with regard to this KIP?
-Matthias
On 11/5/17 6:11 PM, vito jeng wrote:
> Thanks, Guozhang and Matthias. Your comments very useful for me.
>
> I'll update KIP and keep going on.
>
>
>
> ---
> Vito
>
> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Thanks for the KIP Vito!
>>
>> I agree with what Guozhang said. The original idea of the Jira was, to
>> give different exceptions for different "recovery" strategies to the user.
>>
>> For example, if a store is currently recreated, a user just need to wait
>> and can query the store later. On the other hand, if a store go migrated
>> to another instance, a user needs to rediscover the store instead of a
>> "plain retry".
>>
>> Fatal errors might be a third category.
>>
>> Not sure if there is something else?
>>
>> Anyway, the KIP should contain a section that talks about this ideas and
>> reasoning.
>>
>>
>> -Matthias
>>
>>
>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>> Thanks for writing up the KIP.
>>>
>>> Vito, Matthias: one thing that I wanted to figure out first is what
>>> categories of errors we want to notify the users, if we only wants to
>>> distinguish fatal v.s. retriable then probably we should rename the
>>> proposed StateStoreMigratedException / StateStoreClosedException classes.
>>> And then from there we should list what are the possible internal
>>> exceptions ever thrown in those APIs in the call trace, and which
>>> exceptions should be wrapped to what others, and which ones should be
>>> handled without re-throwing, and which ones should not be wrapped at all
>>> but directly thrown to user's face.
>>>
>>> Guozhang
>>>
>>>
>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'd like to start discuss KIP-216:
>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>>>>
>>>> Please have a look.
>>>> Thanks!
>>>>
>>>> ---
>>>> Vito
>>>>
>>>
>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Thanks, Guozhang and Matthias. Your comments very useful for me.
I'll update KIP and keep going on.
---
Vito
On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for the KIP Vito!
>
> I agree with what Guozhang said. The original idea of the Jira was, to
> give different exceptions for different "recovery" strategies to the user.
>
> For example, if a store is currently recreated, a user just need to wait
> and can query the store later. On the other hand, if a store go migrated
> to another instance, a user needs to rediscover the store instead of a
> "plain retry".
>
> Fatal errors might be a third category.
>
> Not sure if there is something else?
>
> Anyway, the KIP should contain a section that talks about this ideas and
> reasoning.
>
>
> -Matthias
>
>
> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > Thanks for writing up the KIP.
> >
> > Vito, Matthias: one thing that I wanted to figure out first is what
> > categories of errors we want to notify the users, if we only wants to
> > distinguish fatal v.s. retriable then probably we should rename the
> > proposed StateStoreMigratedException / StateStoreClosedException classes.
> > And then from there we should list what are the possible internal
> > exceptions ever thrown in those APIs in the call trace, and which
> > exceptions should be wrapped to what others, and which ones should be
> > handled without re-throwing, and which ones should not be wrapped at all
> > but directly thrown to user's face.
> >
> > Guozhang
> >
> >
> > On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw> wrote:
> >
> >> Hi,
> >>
> >> I'd like to start discuss KIP-216:
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>
> >> Please have a look.
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >
> >
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Thanks Navinder and Matthias.
---
Vito
On Sat, Feb 22, 2020 at 8:12 PM Matthias J. Sax <mj...@apache.org> wrote:
> Good find Vito!
>
> What Navinder says makes sense -- as there is no RC for 2.5.0 yet, I
> took the liberty to do a HOTFIX PR so we can address the issue in 2.5.0
> already.
>
> https://github.com/apache/kafka/pull/8158
>
>
> -Matthias
>
> On 2/21/20 11:11 PM, Navinder Brar wrote:
> > Hi Vito,
> >
> > I checked the code and I think you are right. If a user provides a
> wrong partition there will be NPE at
> tasks.get(keyTaskId).getStore(storeName) as that task is not available
> at this machine.
> >
> > I think we split the line:
>
> https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
> into 2 parts and check tasks.get(keyTaskId) separately. If it is null,
> we can throw an InvalidPartitionException. WDYS?
> >
> > Thanks,
> > Navinder
> >
> >
> > On Saturday, 22 February, 2020, 06:22:14 am IST, Vito Jeng
> <vi...@is-land.com.tw> wrote:
> >
> > Hi, Matthias and Navinder,
> >
> > I have a question about the valid partition in
> > StreamThreadStateStoreProvider.
> >
> > In the StreamThreadStateStoreProvider#createKeyTaskId(storeName,
> partition):
> >
>
> https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L103
> >
> > We pass an integer as partition and then use this partition to create
> > TaskId instance in the topic group while loop. How do we make sure the
> > partition is valid? If we pass a correct storeName and a invalid
> partition
> > into createKeyTaskId() , it still looks can be created a new TaskId and
> > would not throw InvalidStateStorePartitionException.
> >
> > I guess this would cause a NullPointerException at line #62 because this
> > keyTaskId cannot found in the task list.
> >
>
> https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
> >
> > Does this right? or there something wrong with me?
> >
> > ---
> > Vito
> >
> >
> > On Wed, Feb 5, 2020 at 2:53 AM Navinder Brar
> > <na...@yahoo.com.invalid> wrote:
> >
> >> Thanks Vito, for incorporating this. Makes sense.
> >>
> >> -Navinder
> >>
> >>
> >> On Wednesday, February 5, 2020, 12:17 AM, Matthias J. Sax <
> >> mjsax@apache.org> wrote:
> >>
> > Thanks Vito!
> >
> > That makes sense to me.
> >
> >
> > On 2/1/20 11:29 PM, Vito Jeng wrote:
> >>>> Hi, folks,
> >>>>
> >>>> KIP-562(KAFKA-9445) already merged three days ago.
> >>>>
> >>>> I have updated KIP-216 to reflect the KIP-562. The main change is
> >>>> to introduce a new exception `InvalidStateStorePartitionException`,
> >>>> will be thrown when user requested partition not available.
> >>>>
> >>>> Please take a look and any feedback is welcome. Thanks Matthias for
> >>>> the reminder.
> >>>>
> >>>> --- Vito
> >>>>
> >>>>
> >>>> On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng <vi...@is-land.com.tw>
> >>>> wrote:
> >>>>
> >>>>> Got it, thanks Matthias.
> >>>>>
> >>>>> --- Vito
> >>>>>
> >>>>>
> >>>>> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
> >>>>> <ma...@confluent.io> wrote:
> >>>>>
> >>>>>> Thanks Vito.
> >>>>>>
> >>>>>> I am also ok with either name. Just a personal slight
> >>>>>> preference, but not a important.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/21/20 6:52 PM, Vito Jeng wrote:
> >>>>>>> Thanks Matthias.
> >>>>>>>
> >>>>>>> The KIP is about InvalidStateStoreException. I pick
> >>>>>>> `StateStoreNotAvailableException` because it may be more
> >>>>>> intuitive
> >>>>>>> than `StreamsNotRunningException`.
> >>>>>>>
> >>>>>>> No matter which one picked, it's good to me.
> >>>>>>>
> >>>>>>> --- Vito
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax
> >>>>>>> <ma...@confluent.io> wrote:
> >>>>>>>
> >>>>>>>> Thanks for updating the KIP!
> >>>>>>>>
> >>>>>>>> One last comment/question: you kept
> >>>>>>>> `StateStoreNotAvailableException`
> >>>>>> in
> >>>>>>>> favor of `StreamsNotRunningException` (to merge both as
> >>>>>>>> suggested).
> >>>>>>>>
> >>>>>>>> I am wondering, if it might be better to keep
> >>>>>>>> `StreamsNotRunningException` instead of
> >>>>>>>> `StateStoreNotAvailableException`, because this exception
> >>>>>>>> is thrown if Streams is in state PENDING_SHUTDOWN /
> >>>>>>>> NOT_RUNNING / ERROR ?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>> On 1/17/20 9:56 PM, John Roesler wrote:
> >>>>>>>>> Thanks, Vito. I've just cast my vote. -John
> >>>>>>>>>
> >>>>>>>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >>>>>>>>>> Hi, folks,
> >>>>>>>>>>
> >>>>>>>>>> Just update the KIP, please take a look.
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>>
> >>>>>>>>>> --- Vito
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng
> >>>>>>>>>> <vi...@is-land.com.tw>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks Bill, John and Matthias. Glad you guys joined
> >>>>>>>>>>> this
> >>>>>> discussion.
> >>>>>>>>>>> I got a lot out of the discussion.
> >>>>>>>>>>>
> >>>>>>>>>>> I would like to update KIP-216 base on John's
> >>>>>>>>>>> suggestion to remove
> >>>>>> the
> >>>>>>>>>>> category.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --- Vito
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
> >>>>>> matthias@confluent.io
> >>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>> Nevertheless, if we omit the categorization, it’s
> >>>>>>>>>>>>> moot.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ack.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I am fine to remove the middle tier. As John
> >>>>>>>>>>>> pointed out, it might
> >>>>>> be
> >>>>>>>>>>>> weird to have only one concrete exception type per
> >>>>>>>>>>>> category. We can
> >>>>>>>> also
> >>>>>>>>>>>> explain in detail how to handle each exception in
> >>>>>>>>>>>> their JavaDocs.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the updates, the KIP LGTM.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Bill
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
> >>>>>> vvcephei@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Haha, your archive game is on point!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What Matthias said in that email is essentially
> >>>>>>>>>>>>>> what I figured
> >>>>>> was
> >>>>>>>> the
> >>>>>>>>>>>>>> rationale. It makes sense, but the point I was
> >>>>>>>>>>>>>> making is that
> >>>>>> this
> >>>>>>>>>>>> really
> >>>>>>>>>>>>>> doesn’t seem like a good way to structure a
> >>>>>>>>>>>>>> production app. On
> >>>>>> the
> >>>>>>>>>>>> other
> >>>>>>>>>>>>>> hand, considering the exception fatal has a
> >>>>>>>>>>>>>> good chance of
> >>>>>> avoiding
> >>>>>>>> a
> >>>>>>>>>>>>>> frustrating debug session if you just forgot to
> >>>>>>>>>>>>>> call start.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Nevertheless, if we omit the categorization,
> >>>>>>>>>>>>>> it’s moot.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It would be easy to add a categorization layer
> >>>>>>>>>>>>>> later if we want
> >>>>>> it,
> >>>>>>>> but
> >>>>>>>>>>>>>> not very easy to change it if we get it wrong.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for your consideration! -John
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> About `StreamsNotStartedException is strange`
> >>>>>>>>>>>>>>> -- The original idea came from Matthias, two
> >>>>>>>>>>>>>>> years ago. :) You can reference here:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6
> > c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> > <
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> >
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>
> > About omitting the categorization --
> >>>>>>>>>>>>>>> It looks reasonable. I'm fine with omitting
> >>>>>>>>>>>>>>> the categorization
> >>>>>> but
> >>>>>>>> not
> >>>>>>>>>>>>>> very
> >>>>>>>>>>>>>>> sure it is a good choice. Does any other
> >>>>>>>>>>>>>>> folks provide opinion?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi, folks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Just update the KIP-216, please take a look.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng
> >>>>>>>>>>>>>>> <vi...@is-land.com.tw>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi, folks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you suggestion, really appreciate it.
> >>>>>>>>>>>>>>>> :) I understand your concern. I'll merge
> >>>>>> StreamsNotRunningException
> >>>>>>>> and
> >>>>>>>>>>>>>>>> StateStoreNotAvailableException.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John
> >>>>>>>>>>>>>>>> Roesler <
> >>>>>> vvcephei@apache.org
> >>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hey Vito,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yes, thanks for the KIP. Sorry the
> >>>>>>>>>>>>>>>>> discussion has been so
> >>>>>> long.
> >>>>>>>>>>>>>>>>> Hopefully, we can close it out soon.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I agree we can drop
> >>>>>>>>>>>>>>>>> StreamsNotRunningException in favor of
> >>>>>>>>>>>>>>>>> just StateStoreNotAvailableException.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Unfortunately, I have some higher-level
> >>>>>>>>>>>>>>>>> concerns. The value of these exceptions
> >>>>>>>>>>>>>>>>> is that they tell you how to handle the
> >>>>>>>>>>>>>>>>> various situations that can arise while
> >>>>>>>>>>>>>>>>> querying a distributed data store.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Ideally, as a caller, I should be able to
> >>>>>>>>>>>>>>>>> just catch
> >>>>>> "retriable"
> >>>>>>>> or
> >>>>>>>>>>>>>>>>> "fatal" and handle them appropriately.
> >>>>>>>>>>>>>>>>> Otherwise, there's no point in having
> >>>>>>>>>>>>>>>>> categories, and we should just have all
> >>>>>>>>>>>>>>>>> the exceptions extend
> >>>>>>>>>>>>>>>>> InvalidStateStoreException.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Presently, it's not possible to tell from
> >>>>>>>>>>>>>>>>> just the "retriable"/"fatal" distinction
> >>>>>>>>>>>>>>>>> what to do. You can tell from the
> >>>>>>>>>>>>>>>>> descriptions of the various exceptions.
> >>>>>>>>>>>>>>>>> E.g.:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Retriable: * StreamsRebalancingException:
> >>>>>>>>>>>>>>>>> the exact same call should just be
> >>>>>>>>>>>>>>>>> retried until the rebalance is complete *
> >>>>>>>>>>>>>>>>> StateStoreMigratedException: the store
> >>>>>>>>>>>>>>>>> handle is now invalid, so you need to
> >>>>>>>>>>>>>>>>> re-discover the instance and get a new
> >>>>>>>>>>>>>>>>> handle on that instance. In other words,
> >>>>>>>>>>>>>>>>> the query itself may be valid, but the
> >>>>>>>>>>>>>>>>> particular method invocation on this
> >>>>>>>>>>>>>>>>> particular instance has encountered a
> >>>>>>>>>>>>>>>>> fatal exception.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Fatal: * UnknownStateStoreException: this
> >>>>>>>>>>>>>>>>> is truly fatal. No amount of retrying or
> >>>>>>>>>>>>>>>>> re-discovering is going to get you a
> >>>>>>>>>>>>>>>>> handle
> >>>>>>>> on a
> >>>>>>>>>>>>>>>>> store that doesn't exist in the cluster.
> >>>>>>>>>>>>>>>>> * StateStoreNotAvailableException: this
> >>>>>>>>>>>>>>>>> is actually
> >>>>>> recoverable,
> >>>>>>>>>>>>>>>>> since the store might exist in the
> >>>>>>>>>>>>>>>>> cluster, but isn't
> >>>>>>>> available
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>> this particular instance (which is shut
> >>>>>>>>>>>>>>>>> down or whatever).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Personally, I'm not a fan of code
> >>>>>>>>>>>>>>>>> bureaucracy, so I'm 100%
> >>>>>> fine
> >>>>>>>>>>>>>>>>> with omitting the categorization and just
> >>>>>>>>>>>>>>>>> having 5 subclasses of
> >>>>>>>>>>>>>>>>> InvalidStateStoreException. Each of them
> >>>>>>>>>>>>>>>>> would tell you how to handle them, and
> >>>>>>>>>>>>>>>>> it's not too many to really understand
> >>>>>>>>>>>>>>>>> and handle each one.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If you really want to have a middle tier,
> >>>>>>>>>>>>>>>>> I'd recommend: *
> >>>>>>>>>>>>>>>>> RetryableStateStoreException: the exact
> >>>>>>>>>>>>>>>>> same call should be repeated. *
> >>>>>>>>>>>>>>>>> RecoverableStateStoreException: the store
> >>>>>>>>>>>>>>>>> handle should be discarded and the caller
> >>>>>>>>>>>>>>>>> should re-discover the location of the
> >>>>>>>>>>>>>>>>> store and repeat the query on the correct
> >>>>>>>>>>>>>>>>> instance. * FatalStateStoreException: the
> >>>>>>>>>>>>>>>>> query/request is totally invalid and will
> >>>>>>>>>>>>>>>>> never succeed.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> However, attempting to categorize the
> >>>>>>>>>>>>>>>>> proposed exceptions reveals even problems
> >>>>>>>>>>>>>>>>> with this categorization: Retriable: *
> >>>>>>>>>>>>>>>>> StreamsRebalancingException Recoverable:
> >>>>>>>>>>>>>>>>> * StateStoreMigratedException *
> >>>>>>>>>>>>>>>>> StreamsNotRunningException Fatal: *
> >>>>>>>>>>>>>>>>> UnknownStateStoreException
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> But StreamsNotStartedException is
> >>>>>>>>>>>>>>>>> strange... It means that one code path
> >>>>>>>>>>>>>>>>> got a handle on a specific KafkaStreams
> >>>>>>>>>>>>>>>>> object instance and sent it a query
> >>>>>>>>>>>>>>>>> before another code path invoked the
> >>>>>>>>>>>>>>>>> start() method on the exact same object
> >>>>>>>>>>>>>>>>> instance. It seems like the most likely
> >>>>>>>>>>>>>>>>> scenario is that whoever wrote the
> >>>>>>>>>>>>>>>>> program just forgot to call start()
> >>>>>>>>>>>>>>>>> before querying, in which case, retrying
> >>>>>>>>>>>>>>>>> isn't going to help, and a fatal
> >>>>>> exception
> >>>>>>>>>>>>>>>>> is more appropriate. I.e., it sounds like
> >>>>>>>>>>>>>>>>> a "first 15 minutes experience" problem,
> >>>>>>>>>>>>>>>>> and making it fatal would be more
> >>>>>>>>>>>>>>>>> helpful. Even in a production context,
> >>>>>>>>>>>>>>>>> there's no reason not to sequence your
> >>>>>>>>>>>>>>>>> application startup such that you don't
> >>>>>>>>>>>>>>>>> accept queries until after Streams is
> >>>>>>>>>>>>>>>>> started. Thus, I guess I'd categorize it
> >>>>>>>>>>>>>>>>> under "fatal".
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Regardless of whether you make it fatal
> >>>>>>>>>>>>>>>>> or retriable, you'd still have a whole
> >>>>>>>>>>>>>>>>> category with only one exception in it,
> >>>>>>>>>>>>>>>>> and the other two categories only have
> >>>>>>>>>>>>>>>>> two exceptions. Plus, as you pointed out
> >>>>>>>>>>>>>>>>> in the KIP, you can't get all exceptions
> >>>>>>>>>>>>>>>>> in all cases anyway: * store() can only
> >>>>>>>>>>>>>>>>> throw NotStarted, NotRunning, and
> >>>>>>>>>>>>>>>>> Unknown * actual store queries can only
> >>>>>>>>>>>>>>>>> throw Rebalancing, Migrated, and
> >>>>>>>>>>>>>>>>> NotRunning
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thus, in practice also, there are exactly
> >>>>>>>>>>>>>>>>> three categories and also exactly three
> >>>>>>>>>>>>>>>>> exception types. It doesn't seem like
> >>>>>>>>>>>>>>>>> there's a great advantage to the
> >>>>>>>>>>>>>>>>> categories here. To avoid the
> >>>>>>>>>>>>>>>>> categorization problem and also to
> >>>>>>>>>>>>>>>>> clarify what exceptions can actually be
> >>>>>>>>>>>>>>>>> thrown in different circumstances, it
> >>>>>>>>>>>>>>>>> seems like we should just: * get rid of
> >>>>>>>>>>>>>>>>> the middle tier and make all the
> >>>>>>>>>>>>>>>>> exceptions extend
> >>>>>>>>>>>>>>>>> InvalidStateStoreException * drop
> >>>>>>>>>>>>>>>>> StateStoreNotAvailableException in favor
> >>>>>>>>>>>>>>>>> of StreamsNotRunningException * clearly
> >>>>>>>>>>>>>>>>> document on all public methods which
> >>>>>>>>>>>>>>>>> exceptions need to be handled
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> How do you feel about this? Thanks,
> >>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill
> >>>>>>>>>>>>>>>>> Bejeck wrote:
> >>>>>>>>>>>>>>>>>> Thanks for KIP Vito.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Overall the KIP LGTM, but I'd have to
> >>>>>>>>>>>>>>>>>> agree with others on
> >>>>>>>> merging
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Since in both cases, the thread state
> >>>>>>>>>>>>>>>>>> is in
> >>>>>> `PENDING_SHUTDOWN ||
> >>>>>>>>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure
> >>>>>>>>>>>>>>>>>> how we could
> >>>>>> distinguish
> >>>>>>>>>>>>>> when to
> >>>>>>>>>>>>>>>>>> use the different exceptions. Maybe a
> >>>>>>>>>>>>>>>>>> good middle ground would be to have a
> >>>>>>>> detailed
> >>>>>>>>>>>>>>>>>> exception message.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The KIP freeze is close, so I think if
> >>>>>>>>>>>>>>>>>> we can agree on this,
> >>>>>> we
> >>>>>>>> can
> >>>>>>>>>>>>>>>>> wrap up
> >>>>>>>>>>>>>>>>>> the voting soon.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks, Bill
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM
> >>>>>>>>>>>>>>>>>> Matthias J. Sax <
> >>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> It's still unclear to me what the
> >>>>>>>>>>>>>>>>>>> advantage is, to have both
> >>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For both cased, the state is
> >>>>>>>>>>>>>>>>>>> `PENDING_SHUTDOWN /
> >>>>>> NOT_RUNNING /
> >>>>>>>>>>>>>> ERROR`
> >>>>>>>>>>>>>>>>>>> and thus, for a user point of view,
> >>>>>>>>>>>>>>>>>>> why does it matter if
> >>>>>> the
> >>>>>>>>>>>>>> store is
> >>>>>>>>>>>>>>>>>>> closed on not? I don't understand
> >>>>>>>>>>>>>>>>>>> why/how this information
> >>>>>>>> would
> >>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> useful? Do you have a concrete
> >>>>>>>>>>>>>>>>>>> example in mind how a user
> >>>>>> would
> >>>>>>>>>>>>>> react
> >>>>>>>>>>>>>>>>>>> differently to both exceptions?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> @Vinoth: about
> >>>>>>>>>>>>>>>>>>> `StreamsRebalancingException` -- to
> >>>>>>>>>>>>>>>>>>> me, it
> >>>>>> seems
> >>>>>>>>>>>>>> best
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> actually do this on a per-query
> >>>>>>>>>>>>>>>>>>> basis, ie, have an overload
> >>>>>>>>>>>>>>>>>>> `KafkaStreams#store(...)` that takes
> >>>>>>>>>>>>>>>>>>> a boolean flag that
> >>>>>> allow
> >>>>>>>> to
> >>>>>>>>>>>>>>>>>>> _disable_ the exception and opt-in to
> >>>>>>>>>>>>>>>>>>> query a active store
> >>>>>>>> during
> >>>>>>>>>>>>>>>>>>> recovery. However, as KIP-535
> >>>>>>>>>>>>>>>>>>> actually introduces this
> >>>>>> change
> >>>>>>>> in
> >>>>>>>>>>>>>>>>>>> behavior, I think KIP-216 should not
> >>>>>>>>>>>>>>>>>>> cover this, but KIP-535
> >>>>>>>>>>>>>> should be
> >>>>>>>>>>>>>>>>>>> updated. I'll follow up on the other
> >>>>>>>>>>>>>>>>>>> KIP thread to raise
> >>>>>> this
> >>>>>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>> Hi, Matthias & Vinoth,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> What is still unclear to me is,
> >>>>>>>>>>>>>>>>>>>>> what we gain by having
> >>>>>> both
> >>>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`. Both
> >>>>>>>>>>>>>>>>>>>>> exception are thrown when
> >>>>>>>>>>>>>>>>>>>>> KafkaStreams is in state
> >>>>>>>>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a
> >>>>>>>>>>>>>>>>>>>>> user what do I gain to
> >>>>>> know
> >>>>>>>>>>>>>> if the
> >>>>>>>>>>>>>>>>>>>>> state store is closed on not -- I
> >>>>>>>>>>>>>>>>>>>>> can't query it anyway?
> >>>>>>>> Maybe
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Yes, both
> >>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>>>>>>>> are fatal exception. But
> >>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>>>>>>>> is fatal exception
> >>>>>> about
> >>>>>>>>>>>>>> state
> >>>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>> related. I think it would be
> >>>>>>>>>>>>>>>>>>>> helpful that if user need to
> >>>>>> distinguish
> >>>>>>>>>>>>>> these
> >>>>>>>>>>>>>>>>> two
> >>>>>>>>>>>>>>>>>>>> different case to handle it.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I'm not very sure, does that make
> >>>>>>>>>>>>>>>>>>>> sense?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM
> >>>>>>>>>>>>>>>>>>>> Vinoth Chandar <
> >>>>>>>>>>>>>> vinoth@apache.org>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> +1 on merging
> >>>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`,
> >>>>>>>>>>>>>>>>>>>>> both exceptions are
> >>>>>> fatal
> >>>>>>>>>>>>>>>>> anyway. IMO
> >>>>>>>>>>>>>>>>>>>>> its best to have these exceptions
> >>>>>>>>>>>>>>>>>>>>> be about the state store
> >>>>>>>>>>>>>> (and not
> >>>>>>>>>>>>>>>>>>> streams
> >>>>>>>>>>>>>>>>>>>>> state), to easier understanding.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Additionally, KIP-535 allows for
> >>>>>>>>>>>>>>>>>>>>> querying of state stores
> >>>>>> in
> >>>>>>>>>>>>>>>>> rebalancing
> >>>>>>>>>>>>>>>>>>>>> state. So do we need the
> >>>>>>>>>>>>>>>>>>>>> StreamsRebalancingException?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias
> >>>>>>>>>>>>>>>>>>>>> J. Sax" <
> >>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>> Sorry that I dropped the ball
> >>>>>>>>>>>>>>>>>>>>>> on this...
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP.
> >>>>>>>>>>>>>>>>>>>>>> Overall LGTM now. Feel free
> >>>>>> to
> >>>>>>>>>>>>>> start
> >>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> VOTE
> >>>>>>>>>>>>>>>>>>>>>> thread.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> What is still unclear to me is,
> >>>>>>>>>>>>>>>>>>>>>> what we gain by having
> >>>>>> both
> >>>>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`.
> >>>>>>>>>>>>>>>>>>> Both
> >>>>>>>>>>>>>>>>>>>>>> exception are thrown when
> >>>>>>>>>>>>>>>>>>>>>> KafkaStreams is in state
> >>>>>>>>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as
> >>>>>>>>>>>>>>>>>>>>>> a user what do I gain to
> >>>>>> know
> >>>>>>>>>>>>>> if
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> state store is closed on not --
> >>>>>>>>>>>>>>>>>>>>>> I can't query it anyway?
> >>>>>>>>>>>>>> Maybe I
> >>>>>>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply,
> >>>>>>>>>>>>>>>>>>>>>>> thanks for the review.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> > Why is it only thrown if the state is REBALANCING? A
> >>>>>> store
> >>>>>>>>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>>>>>>> migrated during a
> >>>>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
> >>>>>>>>>>>>>>>>>>>>>>>> Streams might
> >>>>>>>> resume
> >>>>>>>>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>>>>>>>> RUNNING state and afterward
> >>>>>>>>>>>>>>>>>>>>>>>> somebody tries to use an
> >>>>>> old
> >>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>>>>>>>> Also, if state is
> >>>>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
> >>>>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
> >>>>>>>>>>>>>>>>>>>>>>>> Hence, I think
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>>>>>>>> does only make sense
> >>>>>> during
> >>>>>>>>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thank you point this, already
> >>>>>>>>>>>>>>>>>>>>>>> updated.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Why do we need to distinguish
> >>>>>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> > `KafkaStreamsNotRunningException` may be caused by
> >>>>>> various
> >>>>>>>>>>>>>>>>> reasons, I
> >>>>>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>> it would be helpful that the
> >>>>>>>>>>>>>>>>>>>>>>> user can distinguish whether
> >>>>>>>>>>>>>>>>>>>>>>> it is caused by the state
> >>>>>>>> store
> >>>>>>>>>>>>>>>>> closed.
> >>>>>>>>>>>>>>>>>>>>>>> (Maybe I am wrong...)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Last, why do we distinguish
> >>>>>>>>>>>>>>>>>>>>>>> between `KafkaStreams`
> >>>>>> instance
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it
> >>>>>>>>>>>>>>>>>>>>>>>> seems we should always
> >>>>>> refer to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>>>>>>>> because that is the level
> >>>>>>>>>>>>>>>>>>>>>>>> of granularity in which we
> >>>>>>>>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>>>>>>>> IQ atm.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Totally agree. Do you mean
> >>>>>>>>>>>>>>>>>>>>>>> the naming of state store
> >>>>>>>>>>>>>> exceptions?
> >>>>>>>>>>>>>>>>>>>>>>> I don't have special reason
> >>>>>>>>>>>>>>>>>>>>>>> to distinguish these two.
> >>>>>>>>>>>>>>>>>>>>>>> Your suggestion look more
> >>>>>>>>>>>>>>>>>>>>>>> reasonable for the exception
> >>>>>>>>>>>>>> naming.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Last, for
> >>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
> >>>>>>>>>>>>>>>>>>>>>>> I would add
> >>>>>> that a
> >>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> rediscover the store and
> >>>>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>> store
> >>>>>>>>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>>>>>>>> invalid and a new store
> >>>>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
> >>>>>>>>>>>>>>>>>>>>>>>> That
> >>>>>> is
> >>>>>>>> a
> >>>>>>>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
> >>>>>>>>>>>>>>>>>>>>>>>> that allows for
> >>>>>>>>>>>>>> "blind"
> >>>>>>>>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>>>>>>>> that either resolve (if the
> >>>>>>>>>>>>>>>>>>>>>>>> store is still on the same
> >>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
> >>>>>>>>>>>>>>>>>>>>>>>> changes to
> >>>>>>>>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> store was migrated away
> >>>>>>>>>>>>>>>>>>>>>>>> during rebalancing).
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> The KIP already updated,
> >>>>>>>>>>>>>>>>>>>>>>> please take a look. :)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48
> >>>>>>>>>>>>>>>>>>>>>>> PM Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Any update on this KIP?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM,
> >>>>>>>>>>>>>>>>>>>>>>>> Matthias J. Sax wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply.
> >>>>>>>>>>>>>>>>>>>>>>>>> The 2.4 deadline kept us
> >>>>>> quite
> >>>>>>>>>>>>>> busy.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> > Why is it only thrown if the state is REBALANCING? A
> >>>>>>>> store
> >>>>>>>>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>>>>>>>> migrated during a
> >>>>>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
> >>>>>>>>>>>>>>>>>>>>>>>>> Streams might
> >>>>>>>> resume
> >>>>>>>>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>>>>>>>>> RUNNING state and
> >>>>>>>>>>>>>>>>>>>>>>>>> afterward somebody tries
> >>>>>>>>>>>>>>>>>>>>>>>>> to use an
> >>>>>> old
> >>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>>>>>>>>> Also, if state is
> >>>>>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
> >>>>>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
> >>>>>>>>>>>>>>>>>>>>>>>>> Hence, I think
> >>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>>>>>>>>> does only make sense
> >>>>>> during
> >>>>>>>>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Why do we need to
> >>>>>>>>>>>>>>>>>>>>>>>>> distinguish between
> >>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> > and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Last, why do we
> >>>>>>>>>>>>>>>>>>>>>>>>> distinguish between
> >>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams`
> >>>>>>>>>>>>>> instance and
> >>>>>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me,
> >>>>>>>>>>>>>>>>>>>>>>>>> it seems we should
> >>>>>>>>>>>>>>>>>>>>>>>>> always
> >>>>>> refer
> >>>>>>>> to
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>>>>>>>>> because that is the level
> >>>>>>>>>>>>>>>>>>>>>>>>> of granularity in which
> >>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>>>>>>>> IQ
> >>>>>>>>>>>>>>>>>>>>>>>> atm.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Last, for
> >>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
> >>>>>>>>>>>>>>>>>>>>>>>>> I would add
> >>>>>>>> that a
> >>>>>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>>>> need to
> >>>>>>>>>>>>>>>>>>>>>>>>> rediscover the store and
> >>>>>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
> >>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>> store
> >>>>>>>>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>>>>>>>>> invalid and a new store
> >>>>>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
> >>>>>>>>>>>>>>>>>>>>>>>>> That
> >>>>>>>> is a
> >>>>>>>>>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
> >>>>>>>>>>>>>>>>>>>>>>>>> that allows for
> >>>>>>>>>>>>>> "blind"
> >>>>>>>>>>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>>>>>>>>> that either resolve (if
> >>>>>>>>>>>>>>>>>>>>>>>>> the store is still on the
> >>>>>>>>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>> instance
> >>>>>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
> >>>>>>>>>>>>>>>>>>>>>>>>> changes to
> >>>>>>>>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> store was migrated away
> >>>>>>>>>>>>>>>>>>>>>>>>> during rebalancing).
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito
> >>>>>>>>>>>>>>>>>>>>>>>>> Jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> My bad. The short link
> >>>>>>>>>>>>>>>>>>>>>>>>>> `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> > <https://shorturl.at/CDNT9>
> >>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> > <https://shorturl.at/CDNT9>
> >>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> > <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Please use the
> >>>>>>>>>>>>>>>>>>>>>>>>>> following instead:
> >>>>>>>>>>>>>> https://shorturl.at/bkKQU
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at
> >>>>>>>>>>>>>>>>>>>>>>>>>> 10:53 AM Vito Jeng <
> >>>>>>>>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > Thank you for explanation. I agree with your
> >>>>>> opinion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()`
> >>>>>>>>>>>>>>>>>>>>>>>>>>> would never throw
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > For the case that corresponding thread crashes
> >>>>>> after we
> >>>>>>>>>>>>>>>>> handed out
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> store handle. We may
> >>>>>>>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>>>>> when stream
> >>>>>> thread is
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>> running(
> >>>>>>>>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9)
> >>>>>>>>>>>>>>>>>>>>>>>>>>> or throw
> >>>>>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>>>>> store is
> >>>>>>>>>>>>>>>>>>>>>>>>>>> closed(https://shorturl.at/hrvAN).
> >>>>>>>>>>>>>>>>>>>>>>>>>>> So I
> >>>>>> think
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> do not
> >>>>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> add a new type for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> this case. Does that
> >>>>>>>>>>>>>>>>>>>>>>>>>>> make sense?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> vs
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > I understand your point. I rename
> >>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> > About check unknown state store names:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> hint. I add a new
> >>>>>>>>>>>>>>>>>>>>>>>>>>> type
> >>>>>>>>>>>>>>>>>>>>> `UnknownStateStoreException`
> >>>>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>>> this case.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, we should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> still have fatal
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure why you
> >>>>>>>>>>>>>> remove
> >>>>>>>>>>>>>>>>> it?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you point this,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> already add it
> >>>>>>>>>>>>>>>>>>>>>>>>>>> again.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> The KIP already
> >>>>>>>>>>>>>>>>>>>>>>>>>>> updated, please take
> >>>>>>>>>>>>>>>>>>>>>>>>>>> a look.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>
> >>
> >>
> >>
> >>
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <mj...@apache.org>.
Good find Vito!
What Navinder says makes sense -- as there is no RC for 2.5.0 yet, I
took the liberty to do a HOTFIX PR so we can address the issue in 2.5.0
already.
https://github.com/apache/kafka/pull/8158
-Matthias
On 2/21/20 11:11 PM, Navinder Brar wrote:
> Hi Vito,
>
> I checked the code and I think you are right. If a user provides a
wrong partition there will be NPE at
tasks.get(keyTaskId).getStore(storeName) as that task is not available
at this machine.
>
> I think we split the line:
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
into 2 parts and check tasks.get(keyTaskId) separately. If it is null,
we can throw an InvalidPartitionException. WDYS?
>
> Thanks,
> Navinder
>
>
> On Saturday, 22 February, 2020, 06:22:14 am IST, Vito Jeng
<vi...@is-land.com.tw> wrote:
>
> Hi, Matthias and Navinder,
>
> I have a question about the valid partition in
> StreamThreadStateStoreProvider.
>
> In the StreamThreadStateStoreProvider#createKeyTaskId(storeName,
partition):
>
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L103
>
> We pass an integer as partition and then use this partition to create
> TaskId instance in the topic group while loop. How do we make sure the
> partition is valid? If we pass a correct storeName and a invalid partition
> into createKeyTaskId() , it still looks can be created a new TaskId and
> would not throw InvalidStateStorePartitionException.
>
> I guess this would cause a NullPointerException at line #62 because this
> keyTaskId cannot found in the task list.
>
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
>
> Does this right? or there something wrong with me?
>
> ---
> Vito
>
>
> On Wed, Feb 5, 2020 at 2:53 AM Navinder Brar
> <na...@yahoo.com.invalid> wrote:
>
>> Thanks Vito, for incorporating this. Makes sense.
>>
>> -Navinder
>>
>>
>> On Wednesday, February 5, 2020, 12:17 AM, Matthias J. Sax <
>> mjsax@apache.org> wrote:
>>
> Thanks Vito!
>
> That makes sense to me.
>
>
> On 2/1/20 11:29 PM, Vito Jeng wrote:
>>>> Hi, folks,
>>>>
>>>> KIP-562(KAFKA-9445) already merged three days ago.
>>>>
>>>> I have updated KIP-216 to reflect the KIP-562. The main change is
>>>> to introduce a new exception `InvalidStateStorePartitionException`,
>>>> will be thrown when user requested partition not available.
>>>>
>>>> Please take a look and any feedback is welcome. Thanks Matthias for
>>>> the reminder.
>>>>
>>>> --- Vito
>>>>
>>>>
>>>> On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng <vi...@is-land.com.tw>
>>>> wrote:
>>>>
>>>>> Got it, thanks Matthias.
>>>>>
>>>>> --- Vito
>>>>>
>>>>>
>>>>> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
>>>>> <ma...@confluent.io> wrote:
>>>>>
>>>>>> Thanks Vito.
>>>>>>
>>>>>> I am also ok with either name. Just a personal slight
>>>>>> preference, but not a important.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>> On 1/21/20 6:52 PM, Vito Jeng wrote:
>>>>>>> Thanks Matthias.
>>>>>>>
>>>>>>> The KIP is about InvalidStateStoreException. I pick
>>>>>>> `StateStoreNotAvailableException` because it may be more
>>>>>> intuitive
>>>>>>> than `StreamsNotRunningException`.
>>>>>>>
>>>>>>> No matter which one picked, it's good to me.
>>>>>>>
>>>>>>> --- Vito
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax
>>>>>>> <ma...@confluent.io> wrote:
>>>>>>>
>>>>>>>> Thanks for updating the KIP!
>>>>>>>>
>>>>>>>> One last comment/question: you kept
>>>>>>>> `StateStoreNotAvailableException`
>>>>>> in
>>>>>>>> favor of `StreamsNotRunningException` (to merge both as
>>>>>>>> suggested).
>>>>>>>>
>>>>>>>> I am wondering, if it might be better to keep
>>>>>>>> `StreamsNotRunningException` instead of
>>>>>>>> `StateStoreNotAvailableException`, because this exception
>>>>>>>> is thrown if Streams is in state PENDING_SHUTDOWN /
>>>>>>>> NOT_RUNNING / ERROR ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>> On 1/17/20 9:56 PM, John Roesler wrote:
>>>>>>>>> Thanks, Vito. I've just cast my vote. -John
>>>>>>>>>
>>>>>>>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>>>>>>>>>> Hi, folks,
>>>>>>>>>>
>>>>>>>>>> Just update the KIP, please take a look.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> --- Vito
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng
>>>>>>>>>> <vi...@is-land.com.tw>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Bill, John and Matthias. Glad you guys joined
>>>>>>>>>>> this
>>>>>> discussion.
>>>>>>>>>>> I got a lot out of the discussion.
>>>>>>>>>>>
>>>>>>>>>>> I would like to update KIP-216 base on John's
>>>>>>>>>>> suggestion to remove
>>>>>> the
>>>>>>>>>>> category.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --- Vito
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
>>>>>> matthias@confluent.io
>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> Nevertheless, if we omit the categorization, it’s
>>>>>>>>>>>>> moot.
>>>>>>>>>>>>
>>>>>>>>>>>> Ack.
>>>>>>>>>>>>
>>>>>>>>>>>> I am fine to remove the middle tier. As John
>>>>>>>>>>>> pointed out, it might
>>>>>> be
>>>>>>>>>>>> weird to have only one concrete exception type per
>>>>>>>>>>>> category. We can
>>>>>>>> also
>>>>>>>>>>>> explain in detail how to handle each exception in
>>>>>>>>>>>> their JavaDocs.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>
>>>>>>>>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the updates, the KIP LGTM.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Bill
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
>>>>>> vvcephei@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Haha, your archive game is on point!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What Matthias said in that email is essentially
>>>>>>>>>>>>>> what I figured
>>>>>> was
>>>>>>>> the
>>>>>>>>>>>>>> rationale. It makes sense, but the point I was
>>>>>>>>>>>>>> making is that
>>>>>> this
>>>>>>>>>>>> really
>>>>>>>>>>>>>> doesn’t seem like a good way to structure a
>>>>>>>>>>>>>> production app. On
>>>>>> the
>>>>>>>>>>>> other
>>>>>>>>>>>>>> hand, considering the exception fatal has a
>>>>>>>>>>>>>> good chance of
>>>>>> avoiding
>>>>>>>> a
>>>>>>>>>>>>>> frustrating debug session if you just forgot to
>>>>>>>>>>>>>> call start.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nevertheless, if we omit the categorization,
>>>>>>>>>>>>>> it’s moot.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It would be easy to add a categorization layer
>>>>>>>>>>>>>> later if we want
>>>>>> it,
>>>>>>>> but
>>>>>>>>>>>>>> not very easy to change it if we get it wrong.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for your consideration! -John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> About `StreamsNotStartedException is strange`
>>>>>>>>>>>>>>> -- The original idea came from Matthias, two
>>>>>>>>>>>>>>> years ago. :) You can reference here:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6
> c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> <https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>
> About omitting the categorization --
>>>>>>>>>>>>>>> It looks reasonable. I'm fine with omitting
>>>>>>>>>>>>>>> the categorization
>>>>>> but
>>>>>>>> not
>>>>>>>>>>>>>> very
>>>>>>>>>>>>>>> sure it is a good choice. Does any other
>>>>>>>>>>>>>>> folks provide opinion?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi, folks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Just update the KIP-216, please take a look.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng
>>>>>>>>>>>>>>> <vi...@is-land.com.tw>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi, folks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you suggestion, really appreciate it.
>>>>>>>>>>>>>>>> :) I understand your concern. I'll merge
>>>>>> StreamsNotRunningException
>>>>>>>> and
>>>>>>>>>>>>>>>> StateStoreNotAvailableException.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John
>>>>>>>>>>>>>>>> Roesler <
>>>>>> vvcephei@apache.org
>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hey Vito,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, thanks for the KIP. Sorry the
>>>>>>>>>>>>>>>>> discussion has been so
>>>>>> long.
>>>>>>>>>>>>>>>>> Hopefully, we can close it out soon.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I agree we can drop
>>>>>>>>>>>>>>>>> StreamsNotRunningException in favor of
>>>>>>>>>>>>>>>>> just StateStoreNotAvailableException.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Unfortunately, I have some higher-level
>>>>>>>>>>>>>>>>> concerns. The value of these exceptions
>>>>>>>>>>>>>>>>> is that they tell you how to handle the
>>>>>>>>>>>>>>>>> various situations that can arise while
>>>>>>>>>>>>>>>>> querying a distributed data store.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ideally, as a caller, I should be able to
>>>>>>>>>>>>>>>>> just catch
>>>>>> "retriable"
>>>>>>>> or
>>>>>>>>>>>>>>>>> "fatal" and handle them appropriately.
>>>>>>>>>>>>>>>>> Otherwise, there's no point in having
>>>>>>>>>>>>>>>>> categories, and we should just have all
>>>>>>>>>>>>>>>>> the exceptions extend
>>>>>>>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Presently, it's not possible to tell from
>>>>>>>>>>>>>>>>> just the "retriable"/"fatal" distinction
>>>>>>>>>>>>>>>>> what to do. You can tell from the
>>>>>>>>>>>>>>>>> descriptions of the various exceptions.
>>>>>>>>>>>>>>>>> E.g.:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Retriable: * StreamsRebalancingException:
>>>>>>>>>>>>>>>>> the exact same call should just be
>>>>>>>>>>>>>>>>> retried until the rebalance is complete *
>>>>>>>>>>>>>>>>> StateStoreMigratedException: the store
>>>>>>>>>>>>>>>>> handle is now invalid, so you need to
>>>>>>>>>>>>>>>>> re-discover the instance and get a new
>>>>>>>>>>>>>>>>> handle on that instance. In other words,
>>>>>>>>>>>>>>>>> the query itself may be valid, but the
>>>>>>>>>>>>>>>>> particular method invocation on this
>>>>>>>>>>>>>>>>> particular instance has encountered a
>>>>>>>>>>>>>>>>> fatal exception.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Fatal: * UnknownStateStoreException: this
>>>>>>>>>>>>>>>>> is truly fatal. No amount of retrying or
>>>>>>>>>>>>>>>>> re-discovering is going to get you a
>>>>>>>>>>>>>>>>> handle
>>>>>>>> on a
>>>>>>>>>>>>>>>>> store that doesn't exist in the cluster.
>>>>>>>>>>>>>>>>> * StateStoreNotAvailableException: this
>>>>>>>>>>>>>>>>> is actually
>>>>>> recoverable,
>>>>>>>>>>>>>>>>> since the store might exist in the
>>>>>>>>>>>>>>>>> cluster, but isn't
>>>>>>>> available
>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>> this particular instance (which is shut
>>>>>>>>>>>>>>>>> down or whatever).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Personally, I'm not a fan of code
>>>>>>>>>>>>>>>>> bureaucracy, so I'm 100%
>>>>>> fine
>>>>>>>>>>>>>>>>> with omitting the categorization and just
>>>>>>>>>>>>>>>>> having 5 subclasses of
>>>>>>>>>>>>>>>>> InvalidStateStoreException. Each of them
>>>>>>>>>>>>>>>>> would tell you how to handle them, and
>>>>>>>>>>>>>>>>> it's not too many to really understand
>>>>>>>>>>>>>>>>> and handle each one.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If you really want to have a middle tier,
>>>>>>>>>>>>>>>>> I'd recommend: *
>>>>>>>>>>>>>>>>> RetryableStateStoreException: the exact
>>>>>>>>>>>>>>>>> same call should be repeated. *
>>>>>>>>>>>>>>>>> RecoverableStateStoreException: the store
>>>>>>>>>>>>>>>>> handle should be discarded and the caller
>>>>>>>>>>>>>>>>> should re-discover the location of the
>>>>>>>>>>>>>>>>> store and repeat the query on the correct
>>>>>>>>>>>>>>>>> instance. * FatalStateStoreException: the
>>>>>>>>>>>>>>>>> query/request is totally invalid and will
>>>>>>>>>>>>>>>>> never succeed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, attempting to categorize the
>>>>>>>>>>>>>>>>> proposed exceptions reveals even problems
>>>>>>>>>>>>>>>>> with this categorization: Retriable: *
>>>>>>>>>>>>>>>>> StreamsRebalancingException Recoverable:
>>>>>>>>>>>>>>>>> * StateStoreMigratedException *
>>>>>>>>>>>>>>>>> StreamsNotRunningException Fatal: *
>>>>>>>>>>>>>>>>> UnknownStateStoreException
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But StreamsNotStartedException is
>>>>>>>>>>>>>>>>> strange... It means that one code path
>>>>>>>>>>>>>>>>> got a handle on a specific KafkaStreams
>>>>>>>>>>>>>>>>> object instance and sent it a query
>>>>>>>>>>>>>>>>> before another code path invoked the
>>>>>>>>>>>>>>>>> start() method on the exact same object
>>>>>>>>>>>>>>>>> instance. It seems like the most likely
>>>>>>>>>>>>>>>>> scenario is that whoever wrote the
>>>>>>>>>>>>>>>>> program just forgot to call start()
>>>>>>>>>>>>>>>>> before querying, in which case, retrying
>>>>>>>>>>>>>>>>> isn't going to help, and a fatal
>>>>>> exception
>>>>>>>>>>>>>>>>> is more appropriate. I.e., it sounds like
>>>>>>>>>>>>>>>>> a "first 15 minutes experience" problem,
>>>>>>>>>>>>>>>>> and making it fatal would be more
>>>>>>>>>>>>>>>>> helpful. Even in a production context,
>>>>>>>>>>>>>>>>> there's no reason not to sequence your
>>>>>>>>>>>>>>>>> application startup such that you don't
>>>>>>>>>>>>>>>>> accept queries until after Streams is
>>>>>>>>>>>>>>>>> started. Thus, I guess I'd categorize it
>>>>>>>>>>>>>>>>> under "fatal".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regardless of whether you make it fatal
>>>>>>>>>>>>>>>>> or retriable, you'd still have a whole
>>>>>>>>>>>>>>>>> category with only one exception in it,
>>>>>>>>>>>>>>>>> and the other two categories only have
>>>>>>>>>>>>>>>>> two exceptions. Plus, as you pointed out
>>>>>>>>>>>>>>>>> in the KIP, you can't get all exceptions
>>>>>>>>>>>>>>>>> in all cases anyway: * store() can only
>>>>>>>>>>>>>>>>> throw NotStarted, NotRunning, and
>>>>>>>>>>>>>>>>> Unknown * actual store queries can only
>>>>>>>>>>>>>>>>> throw Rebalancing, Migrated, and
>>>>>>>>>>>>>>>>> NotRunning
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thus, in practice also, there are exactly
>>>>>>>>>>>>>>>>> three categories and also exactly three
>>>>>>>>>>>>>>>>> exception types. It doesn't seem like
>>>>>>>>>>>>>>>>> there's a great advantage to the
>>>>>>>>>>>>>>>>> categories here. To avoid the
>>>>>>>>>>>>>>>>> categorization problem and also to
>>>>>>>>>>>>>>>>> clarify what exceptions can actually be
>>>>>>>>>>>>>>>>> thrown in different circumstances, it
>>>>>>>>>>>>>>>>> seems like we should just: * get rid of
>>>>>>>>>>>>>>>>> the middle tier and make all the
>>>>>>>>>>>>>>>>> exceptions extend
>>>>>>>>>>>>>>>>> InvalidStateStoreException * drop
>>>>>>>>>>>>>>>>> StateStoreNotAvailableException in favor
>>>>>>>>>>>>>>>>> of StreamsNotRunningException * clearly
>>>>>>>>>>>>>>>>> document on all public methods which
>>>>>>>>>>>>>>>>> exceptions need to be handled
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How do you feel about this? Thanks,
>>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill
>>>>>>>>>>>>>>>>> Bejeck wrote:
>>>>>>>>>>>>>>>>>> Thanks for KIP Vito.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Overall the KIP LGTM, but I'd have to
>>>>>>>>>>>>>>>>>> agree with others on
>>>>>>>> merging
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Since in both cases, the thread state
>>>>>>>>>>>>>>>>>> is in
>>>>>> `PENDING_SHUTDOWN ||
>>>>>>>>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure
>>>>>>>>>>>>>>>>>> how we could
>>>>>> distinguish
>>>>>>>>>>>>>> when to
>>>>>>>>>>>>>>>>>> use the different exceptions. Maybe a
>>>>>>>>>>>>>>>>>> good middle ground would be to have a
>>>>>>>> detailed
>>>>>>>>>>>>>>>>>> exception message.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The KIP freeze is close, so I think if
>>>>>>>>>>>>>>>>>> we can agree on this,
>>>>>> we
>>>>>>>> can
>>>>>>>>>>>>>>>>> wrap up
>>>>>>>>>>>>>>>>>> the voting soon.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks, Bill
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM
>>>>>>>>>>>>>>>>>> Matthias J. Sax <
>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It's still unclear to me what the
>>>>>>>>>>>>>>>>>>> advantage is, to have both
>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For both cased, the state is
>>>>>>>>>>>>>>>>>>> `PENDING_SHUTDOWN /
>>>>>> NOT_RUNNING /
>>>>>>>>>>>>>> ERROR`
>>>>>>>>>>>>>>>>>>> and thus, for a user point of view,
>>>>>>>>>>>>>>>>>>> why does it matter if
>>>>>> the
>>>>>>>>>>>>>> store is
>>>>>>>>>>>>>>>>>>> closed on not? I don't understand
>>>>>>>>>>>>>>>>>>> why/how this information
>>>>>>>> would
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> useful? Do you have a concrete
>>>>>>>>>>>>>>>>>>> example in mind how a user
>>>>>> would
>>>>>>>>>>>>>> react
>>>>>>>>>>>>>>>>>>> differently to both exceptions?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @Vinoth: about
>>>>>>>>>>>>>>>>>>> `StreamsRebalancingException` -- to
>>>>>>>>>>>>>>>>>>> me, it
>>>>>> seems
>>>>>>>>>>>>>> best
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> actually do this on a per-query
>>>>>>>>>>>>>>>>>>> basis, ie, have an overload
>>>>>>>>>>>>>>>>>>> `KafkaStreams#store(...)` that takes
>>>>>>>>>>>>>>>>>>> a boolean flag that
>>>>>> allow
>>>>>>>> to
>>>>>>>>>>>>>>>>>>> _disable_ the exception and opt-in to
>>>>>>>>>>>>>>>>>>> query a active store
>>>>>>>> during
>>>>>>>>>>>>>>>>>>> recovery. However, as KIP-535
>>>>>>>>>>>>>>>>>>> actually introduces this
>>>>>> change
>>>>>>>> in
>>>>>>>>>>>>>>>>>>> behavior, I think KIP-216 should not
>>>>>>>>>>>>>>>>>>> cover this, but KIP-535
>>>>>>>>>>>>>> should be
>>>>>>>>>>>>>>>>>>> updated. I'll follow up on the other
>>>>>>>>>>>>>>>>>>> KIP thread to raise
>>>>>> this
>>>>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Hi, Matthias & Vinoth,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> What is still unclear to me is,
>>>>>>>>>>>>>>>>>>>>> what we gain by having
>>>>>> both
>>>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`. Both
>>>>>>>>>>>>>>>>>>>>> exception are thrown when
>>>>>>>>>>>>>>>>>>>>> KafkaStreams is in state
>>>>>>>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a
>>>>>>>>>>>>>>>>>>>>> user what do I gain to
>>>>>> know
>>>>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>>>>> state store is closed on not -- I
>>>>>>>>>>>>>>>>>>>>> can't query it anyway?
>>>>>>>> Maybe
>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Yes, both
>>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>>>>>>>> are fatal exception. But
>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>>>>>>>> is fatal exception
>>>>>> about
>>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>> related. I think it would be
>>>>>>>>>>>>>>>>>>>> helpful that if user need to
>>>>>> distinguish
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>>>>> two
>>>>>>>>>>>>>>>>>>>> different case to handle it.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I'm not very sure, does that make
>>>>>>>>>>>>>>>>>>>> sense?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM
>>>>>>>>>>>>>>>>>>>> Vinoth Chandar <
>>>>>>>>>>>>>> vinoth@apache.org>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> +1 on merging
>>>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`,
>>>>>>>>>>>>>>>>>>>>> both exceptions are
>>>>>> fatal
>>>>>>>>>>>>>>>>> anyway. IMO
>>>>>>>>>>>>>>>>>>>>> its best to have these exceptions
>>>>>>>>>>>>>>>>>>>>> be about the state store
>>>>>>>>>>>>>> (and not
>>>>>>>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>>>>>>>> state), to easier understanding.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Additionally, KIP-535 allows for
>>>>>>>>>>>>>>>>>>>>> querying of state stores
>>>>>> in
>>>>>>>>>>>>>>>>> rebalancing
>>>>>>>>>>>>>>>>>>>>> state. So do we need the
>>>>>>>>>>>>>>>>>>>>> StreamsRebalancingException?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias
>>>>>>>>>>>>>>>>>>>>> J. Sax" <
>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> Sorry that I dropped the ball
>>>>>>>>>>>>>>>>>>>>>> on this...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP.
>>>>>>>>>>>>>>>>>>>>>> Overall LGTM now. Feel free
>>>>>> to
>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> VOTE
>>>>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> What is still unclear to me is,
>>>>>>>>>>>>>>>>>>>>>> what we gain by having
>>>>>> both
>>>>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`.
>>>>>>>>>>>>>>>>>>> Both
>>>>>>>>>>>>>>>>>>>>>> exception are thrown when
>>>>>>>>>>>>>>>>>>>>>> KafkaStreams is in state
>>>>>>>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as
>>>>>>>>>>>>>>>>>>>>>> a user what do I gain to
>>>>>> know
>>>>>>>>>>>>>> if
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> state store is closed on not --
>>>>>>>>>>>>>>>>>>>>>> I can't query it anyway?
>>>>>>>>>>>>>> Maybe I
>>>>>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply,
>>>>>>>>>>>>>>>>>>>>>>> thanks for the review.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
> Why is it only thrown if the state is REBALANCING? A
>>>>>> store
>>>>>>>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>>>>>> migrated during a
>>>>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
>>>>>>>>>>>>>>>>>>>>>>>> Streams might
>>>>>>>> resume
>>>>>>>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>>>>>>> RUNNING state and afterward
>>>>>>>>>>>>>>>>>>>>>>>> somebody tries to use an
>>>>>> old
>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>>>>>>>> Also, if state is
>>>>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
>>>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
>>>>>>>>>>>>>>>>>>>>>>>> Hence, I think
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>>>>>>>> does only make sense
>>>>>> during
>>>>>>>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thank you point this, already
>>>>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Why do we need to distinguish
>>>>>>>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
> `KafkaStreamsNotRunningException` may be caused by
>>>>>> various
>>>>>>>>>>>>>>>>> reasons, I
>>>>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>>>>> it would be helpful that the
>>>>>>>>>>>>>>>>>>>>>>> user can distinguish whether
>>>>>>>>>>>>>>>>>>>>>>> it is caused by the state
>>>>>>>> store
>>>>>>>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>>>>>>>> (Maybe I am wrong...)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Last, why do we distinguish
>>>>>>>>>>>>>>>>>>>>>>> between `KafkaStreams`
>>>>>> instance
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it
>>>>>>>>>>>>>>>>>>>>>>>> seems we should always
>>>>>> refer to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>>>>>>>> because that is the level
>>>>>>>>>>>>>>>>>>>>>>>> of granularity in which we
>>>>>>>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>>>>>>>> IQ atm.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Totally agree. Do you mean
>>>>>>>>>>>>>>>>>>>>>>> the naming of state store
>>>>>>>>>>>>>> exceptions?
>>>>>>>>>>>>>>>>>>>>>>> I don't have special reason
>>>>>>>>>>>>>>>>>>>>>>> to distinguish these two.
>>>>>>>>>>>>>>>>>>>>>>> Your suggestion look more
>>>>>>>>>>>>>>>>>>>>>>> reasonable for the exception
>>>>>>>>>>>>>> naming.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Last, for
>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
>>>>>>>>>>>>>>>>>>>>>>> I would add
>>>>>> that a
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> rediscover the store and
>>>>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>> store
>>>>>>>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>>>>>>>> invalid and a new store
>>>>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
>>>>>>>>>>>>>>>>>>>>>>>> That
>>>>>> is
>>>>>>>> a
>>>>>>>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
>>>>>>>>>>>>>>>>>>>>>>>> that allows for
>>>>>>>>>>>>>> "blind"
>>>>>>>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>>>>>>>> that either resolve (if the
>>>>>>>>>>>>>>>>>>>>>>>> store is still on the same
>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
>>>>>>>>>>>>>>>>>>>>>>>> changes to
>>>>>>>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> store was migrated away
>>>>>>>>>>>>>>>>>>>>>>>> during rebalancing).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The KIP already updated,
>>>>>>>>>>>>>>>>>>>>>>> please take a look. :)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48
>>>>>>>>>>>>>>>>>>>>>>> PM Matthias J. Sax <
>>>>>>>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Any update on this KIP?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM,
>>>>>>>>>>>>>>>>>>>>>>>> Matthias J. Sax wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply.
>>>>>>>>>>>>>>>>>>>>>>>>> The 2.4 deadline kept us
>>>>>> quite
>>>>>>>>>>>>>> busy.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
> Why is it only thrown if the state is REBALANCING? A
>>>>>>>> store
>>>>>>>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>>>>>>> migrated during a
>>>>>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
>>>>>>>>>>>>>>>>>>>>>>>>> Streams might
>>>>>>>> resume
>>>>>>>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>>>>>>>> RUNNING state and
>>>>>>>>>>>>>>>>>>>>>>>>> afterward somebody tries
>>>>>>>>>>>>>>>>>>>>>>>>> to use an
>>>>>> old
>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>>>>>>>>> Also, if state is
>>>>>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
>>>>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
>>>>>>>>>>>>>>>>>>>>>>>>> Hence, I think
>>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>>>>>>>>> does only make sense
>>>>>> during
>>>>>>>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Why do we need to
>>>>>>>>>>>>>>>>>>>>>>>>> distinguish between
>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
> and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Last, why do we
>>>>>>>>>>>>>>>>>>>>>>>>> distinguish between
>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams`
>>>>>>>>>>>>>> instance and
>>>>>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me,
>>>>>>>>>>>>>>>>>>>>>>>>> it seems we should
>>>>>>>>>>>>>>>>>>>>>>>>> always
>>>>>> refer
>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>>>>>>>>> because that is the level
>>>>>>>>>>>>>>>>>>>>>>>>> of granularity in which
>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>>>>>>>> IQ
>>>>>>>>>>>>>>>>>>>>>>>> atm.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Last, for
>>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
>>>>>>>>>>>>>>>>>>>>>>>>> I would add
>>>>>>>> that a
>>>>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>>>>> rediscover the store and
>>>>>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>> store
>>>>>>>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>>>>>>>>> invalid and a new store
>>>>>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
>>>>>>>>>>>>>>>>>>>>>>>>> That
>>>>>>>> is a
>>>>>>>>>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
>>>>>>>>>>>>>>>>>>>>>>>>> that allows for
>>>>>>>>>>>>>> "blind"
>>>>>>>>>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>>>>>>>>> that either resolve (if
>>>>>>>>>>>>>>>>>>>>>>>>> the store is still on the
>>>>>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
>>>>>>>>>>>>>>>>>>>>>>>>> changes to
>>>>>>>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> store was migrated away
>>>>>>>>>>>>>>>>>>>>>>>>> during rebalancing).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito
>>>>>>>>>>>>>>>>>>>>>>>>> Jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> My bad. The short link
>>>>>>>>>>>>>>>>>>>>>>>>>> `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
> <https://shorturl.at/CDNT9>
>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
> <https://shorturl.at/CDNT9> seems incorrect.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Please use the
>>>>>>>>>>>>>>>>>>>>>>>>>> following instead:
>>>>>>>>>>>>>> https://shorturl.at/bkKQU
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at
>>>>>>>>>>>>>>>>>>>>>>>>>> 10:53 AM Vito Jeng <
>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> Thank you for explanation. I agree with your
>>>>>> opinion.
>>>>>>>>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()`
>>>>>>>>>>>>>>>>>>>>>>>>>>> would never throw
>>>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> For the case that corresponding thread crashes
>>>>>> after we
>>>>>>>>>>>>>>>>> handed out
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> store handle. We may
>>>>>>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> In `StreamThreadStateStoreProvider`, we would throw
>>>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>>>>> when stream
>>>>>> thread is
>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>> running(
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9)
>>>>>>>>>>>>>>>>>>>>>>>>>>> or throw
>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>>>>> store is
>>>>>>>>>>>>>>>>>>>>>>>>>>> closed(https://shorturl.at/hrvAN).
>>>>>>>>>>>>>>>>>>>>>>>>>>> So I
>>>>>> think
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> do not
>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> add a new type for
>>>>>>>>>>>>>>>>>>>>>>>>>>> this case. Does that
>>>>>>>>>>>>>>>>>>>>>>>>>>> make sense?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>>>>>> vs
>>>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> I understand your point. I rename
>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
> About check unknown state store names:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
>>>>>>>>>>>>>>>>>>>>>>>>>>> hint. I add a new
>>>>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>>>>> `UnknownStateStoreException`
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>>> this case.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Also, we should
>>>>>>>>>>>>>>>>>>>>>>>>>>>> still have fatal
>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure why you
>>>>>>>>>>>>>> remove
>>>>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you point this,
>>>>>>>>>>>>>>>>>>>>>>>>>>> already add it
>>>>>>>>>>>>>>>>>>>>>>>>>>> again.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> The KIP already
>>>>>>>>>>>>>>>>>>>>>>>>>>> updated, please take
>>>>>>>>>>>>>>>>>>>>>>>>>>> a look.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>>
>>
>>
>>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Navinder Brar <na...@yahoo.com.INVALID>.
Hi Vito,
I checked the code and I think you are right. If a user provides a wrong partition there will be NPE at tasks.get(keyTaskId).getStore(storeName) as that task is not available at this machine.
I think we split the line: https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62 into 2 parts and check tasks.get(keyTaskId) separately. If it is null, we can throw an InvalidPartitionException. WDYS?
Thanks,
Navinder
On Saturday, 22 February, 2020, 06:22:14 am IST, Vito Jeng <vi...@is-land.com.tw> wrote:
Hi, Matthias and Navinder,
I have a question about the valid partition in
StreamThreadStateStoreProvider.
In the StreamThreadStateStoreProvider#createKeyTaskId(storeName, partition):
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L103
We pass an integer as partition and then use this partition to create
TaskId instance in the topic group while loop. How do we make sure the
partition is valid? If we pass a correct storeName and a invalid partition
into createKeyTaskId() , it still looks can be created a new TaskId and
would not throw InvalidStateStorePartitionException.
I guess this would cause a NullPointerException at line #62 because this
keyTaskId cannot found in the task list.
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
Does this right? or there something wrong with me?
---
Vito
On Wed, Feb 5, 2020 at 2:53 AM Navinder Brar
<na...@yahoo.com.invalid> wrote:
> Thanks Vito, for incorporating this. Makes sense.
>
> -Navinder
>
>
> On Wednesday, February 5, 2020, 12:17 AM, Matthias J. Sax <
> mjsax@apache.org> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Thanks Vito!
>
> That makes sense to me.
>
>
> On 2/1/20 11:29 PM, Vito Jeng wrote:
> > Hi, folks,
> >
> > KIP-562(KAFKA-9445) already merged three days ago.
> >
> > I have updated KIP-216 to reflect the KIP-562. The main change is
> > to introduce a new exception `InvalidStateStorePartitionException`,
> > will be thrown when user requested partition not available.
> >
> > Please take a look and any feedback is welcome. Thanks Matthias for
> > the reminder.
> >
> > --- Vito
> >
> >
> > On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng <vi...@is-land.com.tw>
> > wrote:
> >
> >> Got it, thanks Matthias.
> >>
> >> --- Vito
> >>
> >>
> >> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
> >> <ma...@confluent.io> wrote:
> >>
> >>> Thanks Vito.
> >>>
> >>> I am also ok with either name. Just a personal slight
> >>> preference, but not a important.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 1/21/20 6:52 PM, Vito Jeng wrote:
> >>>> Thanks Matthias.
> >>>>
> >>>> The KIP is about InvalidStateStoreException. I pick
> >>>> `StateStoreNotAvailableException` because it may be more
> >>> intuitive
> >>>> than `StreamsNotRunningException`.
> >>>>
> >>>> No matter which one picked, it's good to me.
> >>>>
> >>>> --- Vito
> >>>>
> >>>>
> >>>> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax
> >>>> <ma...@confluent.io> wrote:
> >>>>
> >>>>> Thanks for updating the KIP!
> >>>>>
> >>>>> One last comment/question: you kept
> >>>>> `StateStoreNotAvailableException`
> >>> in
> >>>>> favor of `StreamsNotRunningException` (to merge both as
> >>>>> suggested).
> >>>>>
> >>>>> I am wondering, if it might be better to keep
> >>>>> `StreamsNotRunningException` instead of
> >>>>> `StateStoreNotAvailableException`, because this exception
> >>>>> is thrown if Streams is in state PENDING_SHUTDOWN /
> >>>>> NOT_RUNNING / ERROR ?
> >>>>>
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 1/17/20 9:56 PM, John Roesler wrote:
> >>>>>> Thanks, Vito. I've just cast my vote. -John
> >>>>>>
> >>>>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >>>>>>> Hi, folks,
> >>>>>>>
> >>>>>>> Just update the KIP, please take a look.
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> --- Vito
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng
> >>>>>>> <vi...@is-land.com.tw>
> >>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Bill, John and Matthias. Glad you guys joined
> >>>>>>>> this
> >>> discussion.
> >>>>>>>> I got a lot out of the discussion.
> >>>>>>>>
> >>>>>>>> I would like to update KIP-216 base on John's
> >>>>>>>> suggestion to remove
> >>> the
> >>>>>>>> category.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --- Vito
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
> >>> matthias@confluent.io
> >>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>>> Nevertheless, if we omit the categorization, it’s
> >>>>>>>>>> moot.
> >>>>>>>>>
> >>>>>>>>> Ack.
> >>>>>>>>>
> >>>>>>>>> I am fine to remove the middle tier. As John
> >>>>>>>>> pointed out, it might
> >>> be
> >>>>>>>>> weird to have only one concrete exception type per
> >>>>>>>>> category. We can
> >>>>> also
> >>>>>>>>> explain in detail how to handle each exception in
> >>>>>>>>> their JavaDocs.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >>>>>>>>>> Vito,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the updates, the KIP LGTM.
> >>>>>>>>>>
> >>>>>>>>>> -Bill
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
> >>> vvcephei@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>
> >>>>>>>>>>> Haha, your archive game is on point!
> >>>>>>>>>>>
> >>>>>>>>>>> What Matthias said in that email is essentially
> >>>>>>>>>>> what I figured
> >>> was
> >>>>> the
> >>>>>>>>>>> rationale. It makes sense, but the point I was
> >>>>>>>>>>> making is that
> >>> this
> >>>>>>>>> really
> >>>>>>>>>>> doesn’t seem like a good way to structure a
> >>>>>>>>>>> production app. On
> >>> the
> >>>>>>>>> other
> >>>>>>>>>>> hand, considering the exception fatal has a
> >>>>>>>>>>> good chance of
> >>> avoiding
> >>>>> a
> >>>>>>>>>>> frustrating debug session if you just forgot to
> >>>>>>>>>>> call start.
> >>>>>>>>>>>
> >>>>>>>>>>> Nevertheless, if we omit the categorization,
> >>>>>>>>>>> it’s moot.
> >>>>>>>>>>>
> >>>>>>>>>>> It would be easy to add a categorization layer
> >>>>>>>>>>> later if we want
> >>> it,
> >>>>> but
> >>>>>>>>>>> not very easy to change it if we get it wrong.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for your consideration! -John
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>
> >>>>>>>>>>>> About `StreamsNotStartedException is strange`
> >>>>>>>>>>>> -- The original idea came from Matthias, two
> >>>>>>>>>>>> years ago. :) You can reference here:
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6
> c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> <https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>
> About omitting the categorization --
> >>>>>>>>>>>> It looks reasonable. I'm fine with omitting
> >>>>>>>>>>>> the categorization
> >>> but
> >>>>> not
> >>>>>>>>>>> very
> >>>>>>>>>>>> sure it is a good choice. Does any other
> >>>>>>>>>>>> folks provide opinion?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi, folks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Just update the KIP-216, please take a look.
> >>>>>>>>>>>>
> >>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng
> >>>>>>>>>>>> <vi...@is-land.com.tw>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, folks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you suggestion, really appreciate it.
> >>>>>>>>>>>>> :) I understand your concern. I'll merge
> >>> StreamsNotRunningException
> >>>>> and
> >>>>>>>>>>>>> StateStoreNotAvailableException.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John
> >>>>>>>>>>>>> Roesler <
> >>> vvcephei@apache.org
> >>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hey Vito,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes, thanks for the KIP. Sorry the
> >>>>>>>>>>>>>> discussion has been so
> >>> long.
> >>>>>>>>>>>>>> Hopefully, we can close it out soon.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I agree we can drop
> >>>>>>>>>>>>>> StreamsNotRunningException in favor of
> >>>>>>>>>>>>>> just StateStoreNotAvailableException.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Unfortunately, I have some higher-level
> >>>>>>>>>>>>>> concerns. The value of these exceptions
> >>>>>>>>>>>>>> is that they tell you how to handle the
> >>>>>>>>>>>>>> various situations that can arise while
> >>>>>>>>>>>>>> querying a distributed data store.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ideally, as a caller, I should be able to
> >>>>>>>>>>>>>> just catch
> >>> "retriable"
> >>>>> or
> >>>>>>>>>>>>>> "fatal" and handle them appropriately.
> >>>>>>>>>>>>>> Otherwise, there's no point in having
> >>>>>>>>>>>>>> categories, and we should just have all
> >>>>>>>>>>>>>> the exceptions extend
> >>>>>>>>>>>>>> InvalidStateStoreException.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Presently, it's not possible to tell from
> >>>>>>>>>>>>>> just the "retriable"/"fatal" distinction
> >>>>>>>>>>>>>> what to do. You can tell from the
> >>>>>>>>>>>>>> descriptions of the various exceptions.
> >>>>>>>>>>>>>> E.g.:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Retriable: * StreamsRebalancingException:
> >>>>>>>>>>>>>> the exact same call should just be
> >>>>>>>>>>>>>> retried until the rebalance is complete *
> >>>>>>>>>>>>>> StateStoreMigratedException: the store
> >>>>>>>>>>>>>> handle is now invalid, so you need to
> >>>>>>>>>>>>>> re-discover the instance and get a new
> >>>>>>>>>>>>>> handle on that instance. In other words,
> >>>>>>>>>>>>>> the query itself may be valid, but the
> >>>>>>>>>>>>>> particular method invocation on this
> >>>>>>>>>>>>>> particular instance has encountered a
> >>>>>>>>>>>>>> fatal exception.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Fatal: * UnknownStateStoreException: this
> >>>>>>>>>>>>>> is truly fatal. No amount of retrying or
> >>>>>>>>>>>>>> re-discovering is going to get you a
> >>>>>>>>>>>>>> handle
> >>>>> on a
> >>>>>>>>>>>>>> store that doesn't exist in the cluster.
> >>>>>>>>>>>>>> * StateStoreNotAvailableException: this
> >>>>>>>>>>>>>> is actually
> >>> recoverable,
> >>>>>>>>>>>>>> since the store might exist in the
> >>>>>>>>>>>>>> cluster, but isn't
> >>>>> available
> >>>>>>>>> on
> >>>>>>>>>>>>>> this particular instance (which is shut
> >>>>>>>>>>>>>> down or whatever).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Personally, I'm not a fan of code
> >>>>>>>>>>>>>> bureaucracy, so I'm 100%
> >>> fine
> >>>>>>>>>>>>>> with omitting the categorization and just
> >>>>>>>>>>>>>> having 5 subclasses of
> >>>>>>>>>>>>>> InvalidStateStoreException. Each of them
> >>>>>>>>>>>>>> would tell you how to handle them, and
> >>>>>>>>>>>>>> it's not too many to really understand
> >>>>>>>>>>>>>> and handle each one.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If you really want to have a middle tier,
> >>>>>>>>>>>>>> I'd recommend: *
> >>>>>>>>>>>>>> RetryableStateStoreException: the exact
> >>>>>>>>>>>>>> same call should be repeated. *
> >>>>>>>>>>>>>> RecoverableStateStoreException: the store
> >>>>>>>>>>>>>> handle should be discarded and the caller
> >>>>>>>>>>>>>> should re-discover the location of the
> >>>>>>>>>>>>>> store and repeat the query on the correct
> >>>>>>>>>>>>>> instance. * FatalStateStoreException: the
> >>>>>>>>>>>>>> query/request is totally invalid and will
> >>>>>>>>>>>>>> never succeed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> However, attempting to categorize the
> >>>>>>>>>>>>>> proposed exceptions reveals even problems
> >>>>>>>>>>>>>> with this categorization: Retriable: *
> >>>>>>>>>>>>>> StreamsRebalancingException Recoverable:
> >>>>>>>>>>>>>> * StateStoreMigratedException *
> >>>>>>>>>>>>>> StreamsNotRunningException Fatal: *
> >>>>>>>>>>>>>> UnknownStateStoreException
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> But StreamsNotStartedException is
> >>>>>>>>>>>>>> strange... It means that one code path
> >>>>>>>>>>>>>> got a handle on a specific KafkaStreams
> >>>>>>>>>>>>>> object instance and sent it a query
> >>>>>>>>>>>>>> before another code path invoked the
> >>>>>>>>>>>>>> start() method on the exact same object
> >>>>>>>>>>>>>> instance. It seems like the most likely
> >>>>>>>>>>>>>> scenario is that whoever wrote the
> >>>>>>>>>>>>>> program just forgot to call start()
> >>>>>>>>>>>>>> before querying, in which case, retrying
> >>>>>>>>>>>>>> isn't going to help, and a fatal
> >>> exception
> >>>>>>>>>>>>>> is more appropriate. I.e., it sounds like
> >>>>>>>>>>>>>> a "first 15 minutes experience" problem,
> >>>>>>>>>>>>>> and making it fatal would be more
> >>>>>>>>>>>>>> helpful. Even in a production context,
> >>>>>>>>>>>>>> there's no reason not to sequence your
> >>>>>>>>>>>>>> application startup such that you don't
> >>>>>>>>>>>>>> accept queries until after Streams is
> >>>>>>>>>>>>>> started. Thus, I guess I'd categorize it
> >>>>>>>>>>>>>> under "fatal".
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regardless of whether you make it fatal
> >>>>>>>>>>>>>> or retriable, you'd still have a whole
> >>>>>>>>>>>>>> category with only one exception in it,
> >>>>>>>>>>>>>> and the other two categories only have
> >>>>>>>>>>>>>> two exceptions. Plus, as you pointed out
> >>>>>>>>>>>>>> in the KIP, you can't get all exceptions
> >>>>>>>>>>>>>> in all cases anyway: * store() can only
> >>>>>>>>>>>>>> throw NotStarted, NotRunning, and
> >>>>>>>>>>>>>> Unknown * actual store queries can only
> >>>>>>>>>>>>>> throw Rebalancing, Migrated, and
> >>>>>>>>>>>>>> NotRunning
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thus, in practice also, there are exactly
> >>>>>>>>>>>>>> three categories and also exactly three
> >>>>>>>>>>>>>> exception types. It doesn't seem like
> >>>>>>>>>>>>>> there's a great advantage to the
> >>>>>>>>>>>>>> categories here. To avoid the
> >>>>>>>>>>>>>> categorization problem and also to
> >>>>>>>>>>>>>> clarify what exceptions can actually be
> >>>>>>>>>>>>>> thrown in different circumstances, it
> >>>>>>>>>>>>>> seems like we should just: * get rid of
> >>>>>>>>>>>>>> the middle tier and make all the
> >>>>>>>>>>>>>> exceptions extend
> >>>>>>>>>>>>>> InvalidStateStoreException * drop
> >>>>>>>>>>>>>> StateStoreNotAvailableException in favor
> >>>>>>>>>>>>>> of StreamsNotRunningException * clearly
> >>>>>>>>>>>>>> document on all public methods which
> >>>>>>>>>>>>>> exceptions need to be handled
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> How do you feel about this? Thanks,
> >>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill
> >>>>>>>>>>>>>> Bejeck wrote:
> >>>>>>>>>>>>>>> Thanks for KIP Vito.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Overall the KIP LGTM, but I'd have to
> >>>>>>>>>>>>>>> agree with others on
> >>>>> merging
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Since in both cases, the thread state
> >>>>>>>>>>>>>>> is in
> >>> `PENDING_SHUTDOWN ||
> >>>>>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure
> >>>>>>>>>>>>>>> how we could
> >>> distinguish
> >>>>>>>>>>> when to
> >>>>>>>>>>>>>>> use the different exceptions. Maybe a
> >>>>>>>>>>>>>>> good middle ground would be to have a
> >>>>> detailed
> >>>>>>>>>>>>>>> exception message.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The KIP freeze is close, so I think if
> >>>>>>>>>>>>>>> we can agree on this,
> >>> we
> >>>>> can
> >>>>>>>>>>>>>> wrap up
> >>>>>>>>>>>>>>> the voting soon.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, Bill
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM
> >>>>>>>>>>>>>>> Matthias J. Sax <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It's still unclear to me what the
> >>>>>>>>>>>>>>>> advantage is, to have both
> >>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For both cased, the state is
> >>>>>>>>>>>>>>>> `PENDING_SHUTDOWN /
> >>> NOT_RUNNING /
> >>>>>>>>>>> ERROR`
> >>>>>>>>>>>>>>>> and thus, for a user point of view,
> >>>>>>>>>>>>>>>> why does it matter if
> >>> the
> >>>>>>>>>>> store is
> >>>>>>>>>>>>>>>> closed on not? I don't understand
> >>>>>>>>>>>>>>>> why/how this information
> >>>>> would
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>> useful? Do you have a concrete
> >>>>>>>>>>>>>>>> example in mind how a user
> >>> would
> >>>>>>>>>>> react
> >>>>>>>>>>>>>>>> differently to both exceptions?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> @Vinoth: about
> >>>>>>>>>>>>>>>> `StreamsRebalancingException` -- to
> >>>>>>>>>>>>>>>> me, it
> >>> seems
> >>>>>>>>>>> best
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> actually do this on a per-query
> >>>>>>>>>>>>>>>> basis, ie, have an overload
> >>>>>>>>>>>>>>>> `KafkaStreams#store(...)` that takes
> >>>>>>>>>>>>>>>> a boolean flag that
> >>> allow
> >>>>> to
> >>>>>>>>>>>>>>>> _disable_ the exception and opt-in to
> >>>>>>>>>>>>>>>> query a active store
> >>>>> during
> >>>>>>>>>>>>>>>> recovery. However, as KIP-535
> >>>>>>>>>>>>>>>> actually introduces this
> >>> change
> >>>>> in
> >>>>>>>>>>>>>>>> behavior, I think KIP-216 should not
> >>>>>>>>>>>>>>>> cover this, but KIP-535
> >>>>>>>>>>> should be
> >>>>>>>>>>>>>>>> updated. I'll follow up on the other
> >>>>>>>>>>>>>>>> KIP thread to raise
> >>> this
> >>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> Hi, Matthias & Vinoth,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> What is still unclear to me is,
> >>>>>>>>>>>>>>>>>> what we gain by having
> >>> both
> >>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>> `StateStoreNotAvailableException`. Both
> >>>>>>>>>>>>>>>>>> exception are thrown when
> >>>>>>>>>>>>>>>>>> KafkaStreams is in state
> >>>>>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a
> >>>>>>>>>>>>>>>>>> user what do I gain to
> >>> know
> >>>>>>>>>>> if the
> >>>>>>>>>>>>>>>>>> state store is closed on not -- I
> >>>>>>>>>>>>>>>>>> can't query it anyway?
> >>>>> Maybe
> >>>>>>>>>>> I
> >>>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yes, both
> >>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>>>>> are fatal exception. But
> >>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>>>>> is fatal exception
> >>> about
> >>>>>>>>>>> state
> >>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>> related. I think it would be
> >>>>>>>>>>>>>>>>> helpful that if user need to
> >>> distinguish
> >>>>>>>>>>> these
> >>>>>>>>>>>>>> two
> >>>>>>>>>>>>>>>>> different case to handle it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'm not very sure, does that make
> >>>>>>>>>>>>>>>>> sense?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM
> >>>>>>>>>>>>>>>>> Vinoth Chandar <
> >>>>>>>>>>> vinoth@apache.org>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +1 on merging
> >>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`,
> >>>>>>>>>>>>>>>>>> both exceptions are
> >>> fatal
> >>>>>>>>>>>>>> anyway. IMO
> >>>>>>>>>>>>>>>>>> its best to have these exceptions
> >>>>>>>>>>>>>>>>>> be about the state store
> >>>>>>>>>>> (and not
> >>>>>>>>>>>>>>>> streams
> >>>>>>>>>>>>>>>>>> state), to easier understanding.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Additionally, KIP-535 allows for
> >>>>>>>>>>>>>>>>>> querying of state stores
> >>> in
> >>>>>>>>>>>>>> rebalancing
> >>>>>>>>>>>>>>>>>> state. So do we need the
> >>>>>>>>>>>>>>>>>> StreamsRebalancingException?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias
> >>>>>>>>>>>>>>>>>> J. Sax" <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>> Sorry that I dropped the ball
> >>>>>>>>>>>>>>>>>>> on this...
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for updating the KIP.
> >>>>>>>>>>>>>>>>>>> Overall LGTM now. Feel free
> >>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> VOTE
> >>>>>>>>>>>>>>>>>>> thread.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> What is still unclear to me is,
> >>>>>>>>>>>>>>>>>>> what we gain by having
> >>> both
> >>>>>>>>>>>>>>>>>>> `StreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>> `StateStoreNotAvailableException`.
> >>>>>>>>>>>>>>>> Both
> >>>>>>>>>>>>>>>>>>> exception are thrown when
> >>>>>>>>>>>>>>>>>>> KafkaStreams is in state
> >>>>>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as
> >>>>>>>>>>>>>>>>>>> a user what do I gain to
> >>> know
> >>>>>>>>>>> if
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> state store is closed on not --
> >>>>>>>>>>>>>>>>>>> I can't query it anyway?
> >>>>>>>>>>> Maybe I
> >>>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>> Sorry for the late reply,
> >>>>>>>>>>>>>>>>>>>> thanks for the review.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> Why is it only thrown if the state is REBALANCING? A
> >>> store
> >>>>>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>>>> migrated during a
> >>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
> >>>>>>>>>>>>>>>>>>>>> Streams might
> >>>>> resume
> >>>>>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>>>>> RUNNING state and afterward
> >>>>>>>>>>>>>>>>>>>>> somebody tries to use an
> >>> old
> >>>>>>>>>>> store
> >>>>>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>>>>> Also, if state is
> >>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
> >>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
> >>>>>>>>>>>>>>>>>>>>> Hence, I think
> >>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>>>>> does only make sense
> >>> during
> >>>>>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thank you point this, already
> >>>>>>>>>>>>>>>>>>>> updated.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Why do we need to distinguish
> >>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> `KafkaStreamsNotRunningException` may be caused by
> >>> various
> >>>>>>>>>>>>>> reasons, I
> >>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>> it would be helpful that the
> >>>>>>>>>>>>>>>>>>>> user can distinguish whether
> >>>>>>>>>>>>>>>>>>>> it is caused by the state
> >>>>> store
> >>>>>>>>>>>>>> closed.
> >>>>>>>>>>>>>>>>>>>> (Maybe I am wrong...)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Last, why do we distinguish
> >>>>>>>>>>>>>>>>>>>> between `KafkaStreams`
> >>> instance
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it
> >>>>>>>>>>>>>>>>>>>>> seems we should always
> >>> refer to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>>>>> because that is the level
> >>>>>>>>>>>>>>>>>>>>> of granularity in which we
> >>>>>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>>>>> IQ atm.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Totally agree. Do you mean
> >>>>>>>>>>>>>>>>>>>> the naming of state store
> >>>>>>>>>>> exceptions?
> >>>>>>>>>>>>>>>>>>>> I don't have special reason
> >>>>>>>>>>>>>>>>>>>> to distinguish these two.
> >>>>>>>>>>>>>>>>>>>> Your suggestion look more
> >>>>>>>>>>>>>>>>>>>> reasonable for the exception
> >>>>>>>>>>> naming.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Last, for
> >>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
> >>>>>>>>>>>>>>>>>>>> I would add
> >>> that a
> >>>>>>>>>>> user
> >>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> rediscover the store and
> >>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
> >>>>>>>>>>>>>>>>>>>>> the
> >>> store
> >>>>>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>>>>> invalid and a new store
> >>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
> >>>>>>>>>>>>>>>>>>>>> That
> >>> is
> >>>>> a
> >>>>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
> >>>>>>>>>>>>>>>>>>>>> that allows for
> >>>>>>>>>>> "blind"
> >>>>>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>>>>> that either resolve (if the
> >>>>>>>>>>>>>>>>>>>>> store is still on the same
> >>>>>>>>>>> instance
> >>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
> >>>>>>>>>>>>>>>>>>>>> changes to
> >>>>>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> store was migrated away
> >>>>>>>>>>>>>>>>>>>>> during rebalancing).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The KIP already updated,
> >>>>>>>>>>>>>>>>>>>> please take a look. :)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48
> >>>>>>>>>>>>>>>>>>>> PM Matthias J. Sax <
> >>>>>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Any update on this KIP?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM,
> >>>>>>>>>>>>>>>>>>>>> Matthias J. Sax wrote:
> >>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply.
> >>>>>>>>>>>>>>>>>>>>>> The 2.4 deadline kept us
> >>> quite
> >>>>>>>>>>> busy.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> Why is it only thrown if the state is REBALANCING? A
> >>>>> store
> >>>>>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>>>>> migrated during a
> >>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
> >>>>>>>>>>>>>>>>>>>>>> Streams might
> >>>>> resume
> >>>>>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>>>>>> RUNNING state and
> >>>>>>>>>>>>>>>>>>>>>> afterward somebody tries
> >>>>>>>>>>>>>>>>>>>>>> to use an
> >>> old
> >>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>>>>>> Also, if state is
> >>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
> >>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
> >>>>>>>>>>>>>>>>>>>>>> Hence, I think
> >>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>>>>>> does only make sense
> >>> during
> >>>>>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Why do we need to
> >>>>>>>>>>>>>>>>>>>>>> distinguish between
> >>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Last, why do we
> >>>>>>>>>>>>>>>>>>>>>> distinguish between
> >>>>>>>>>>>>>>>>>>>>>> `KafkaStreams`
> >>>>>>>>>>> instance and
> >>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me,
> >>>>>>>>>>>>>>>>>>>>>> it seems we should
> >>>>>>>>>>>>>>>>>>>>>> always
> >>> refer
> >>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>>>>>> because that is the level
> >>>>>>>>>>>>>>>>>>>>>> of granularity in which
> >>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>>>>> IQ
> >>>>>>>>>>>>>>>>>>>>> atm.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Last, for
> >>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
> >>>>>>>>>>>>>>>>>>>>>> I would add
> >>>>> that a
> >>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>> need to
> >>>>>>>>>>>>>>>>>>>>>> rediscover the store and
> >>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>> store
> >>>>>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>>>>>> invalid and a new store
> >>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
> >>>>>>>>>>>>>>>>>>>>>> That
> >>>>> is a
> >>>>>>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
> >>>>>>>>>>>>>>>>>>>>>> that allows for
> >>>>>>>>>>> "blind"
> >>>>>>>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>>>>>> that either resolve (if
> >>>>>>>>>>>>>>>>>>>>>> the store is still on the
> >>>>>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>> instance
> >>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
> >>>>>>>>>>>>>>>>>>>>>> changes to
> >>>>>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> store was migrated away
> >>>>>>>>>>>>>>>>>>>>>> during rebalancing).
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito
> >>>>>>>>>>>>>>>>>>>>>> Jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>> My bad. The short link
> >>>>>>>>>>>>>>>>>>>>>>> `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>
> <https://shorturl.at/CDNT9>
> >>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Please use the
> >>>>>>>>>>>>>>>>>>>>>>> following instead:
> >>>>>>>>>>> https://shorturl.at/bkKQU
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at
> >>>>>>>>>>>>>>>>>>>>>>> 10:53 AM Vito Jeng <
> >>>>>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> Thank you for explanation. I agree with your
> >>> opinion.
> >>>>>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()`
> >>>>>>>>>>>>>>>>>>>>>>>> would never throw
> >>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> For the case that corresponding thread crashes
> >>> after we
> >>>>>>>>>>>>>> handed out
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> store handle. We may
> >>>>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>> when stream
> >>> thread is
> >>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> running(
> >>>>>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9)
> >>>>>>>>>>>>>>>>>>>>>>>> or throw
> >>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>> store is
> >>>>>>>>>>>>>>>>>>>>>>>> closed(https://shorturl.at/hrvAN).
> >>>>>>>>>>>>>>>>>>>>>>>> So I
> >>> think
> >>>>>>>>>>> we
> >>>>>>>>>>>>>> do not
> >>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> add a new type for
> >>>>>>>>>>>>>>>>>>>>>>>> this case. Does that
> >>>>>>>>>>>>>>>>>>>>>>>> make sense?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>>> vs
> >>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> I understand your point. I rename
> >>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> About check unknown state store names:
> >>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> >>>>>>>>>>>>>>>>>>>>>>>> hint. I add a new
> >>>>>>>>>>>>>>>>>>>>>>>> type
> >>>>>>>>>>>>>>>>>> `UnknownStateStoreException`
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>> this case.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Also, we should
> >>>>>>>>>>>>>>>>>>>>>>>>> still have fatal
> >>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>>>> Not sure why you
> >>>>>>>>>>> remove
> >>>>>>>>>>>>>> it?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thank you point this,
> >>>>>>>>>>>>>>>>>>>>>>>> already add it
> >>>>>>>>>>>>>>>>>>>>>>>> again.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The KIP already
> >>>>>>>>>>>>>>>>>>>>>>>> updated, please take
> >>>>>>>>>>>>>>>>>>>>>>>> a look.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl45vCwACgkQO4miYXKq
> /OhT4Q//elvNj2BRIMd7SqyQhpa6sSq4k2iG3wFlr/B2xkmkZPWUICt3SpCisIxG
> RlP5ml7Mi8IcWv9jmMux9C1NpftboLudxIUcun/I4cP0T3S7ytXfzdO+V1WuD9xe
> hfeqMwJTytusEv5VO3MyY+jYGqmPJrPJRViZ2Uwsj3Ojk0SZRb/m+b6ah6BGMh0E
> nlUIux7cabg1OZ/ee0x21hcAB32lVIRfPLKZeHCTHuYmlW76X4pXhFxOm69Pumtj
> ciVMr/pl7B+XiMk0C2Po04zQcP8+5/O7LYU4e8ha12NtuNuDTlQLGUb+S9Qrbxb2
> xdFrlC81RROmIdIOTLiJeeBqhzd19llDgMhBf4spJMOeLn7359PX6r/9tDqonJrQ
> wpSp4S54MDYfTPHUPqA6u/FsE0BX4EMU71ckC9rXVRRUnQ8A8cMk/6qWlRu8NvxQ
> IiYuSo48UENlcTHxXC/rzIDp54gvtv2iml8QcOg6tS+hzhAI4yoY+1w4pnjPmRo0
> 4D1hnq5bd1SLrGSGcCxVbN0jtwgkcr50HBb1UkIDRndqapfKwZOMV65tIwTvxaaZ
> r2QPYyPd6ZDgeMdy0r94z0SMwatqGCmJD9EQmSmLBulemrvYxPGUevQls+WgE4WE
> zc3LtoyfdlXlkttTU0Q/dm2H9OU4s3QO+6PEdC+Qwi3aCKKhdUQ=
> =4EMC
> -----END PGP SIGNATURE-----
>
>
>
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, Matthias and Navinder,
I have a question about the valid partition in
StreamThreadStateStoreProvider.
In the StreamThreadStateStoreProvider#createKeyTaskId(storeName, partition):
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L103
We pass an integer as partition and then use this partition to create
TaskId instance in the topic group while loop. How do we make sure the
partition is valid? If we pass a correct storeName and a invalid partition
into createKeyTaskId() , it still looks can be created a new TaskId and
would not throw InvalidStateStorePartitionException.
I guess this would cause a NullPointerException at line #62 because this
keyTaskId cannot found in the task list.
https://github.com/apache/kafka/blob/bbfecaef725456f648f03530d26a5395042966fa/streams/src/main/java/org/apache/kafka/streams/state/internals/StreamThreadStateStoreProvider.java#L62
Does this right? or there something wrong with me?
---
Vito
On Wed, Feb 5, 2020 at 2:53 AM Navinder Brar
<na...@yahoo.com.invalid> wrote:
> Thanks Vito, for incorporating this. Makes sense.
>
> -Navinder
>
>
> On Wednesday, February 5, 2020, 12:17 AM, Matthias J. Sax <
> mjsax@apache.org> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Thanks Vito!
>
> That makes sense to me.
>
>
> On 2/1/20 11:29 PM, Vito Jeng wrote:
> > Hi, folks,
> >
> > KIP-562(KAFKA-9445) already merged three days ago.
> >
> > I have updated KIP-216 to reflect the KIP-562. The main change is
> > to introduce a new exception `InvalidStateStorePartitionException`,
> > will be thrown when user requested partition not available.
> >
> > Please take a look and any feedback is welcome. Thanks Matthias for
> > the reminder.
> >
> > --- Vito
> >
> >
> > On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng <vi...@is-land.com.tw>
> > wrote:
> >
> >> Got it, thanks Matthias.
> >>
> >> --- Vito
> >>
> >>
> >> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
> >> <ma...@confluent.io> wrote:
> >>
> >>> Thanks Vito.
> >>>
> >>> I am also ok with either name. Just a personal slight
> >>> preference, but not a important.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 1/21/20 6:52 PM, Vito Jeng wrote:
> >>>> Thanks Matthias.
> >>>>
> >>>> The KIP is about InvalidStateStoreException. I pick
> >>>> `StateStoreNotAvailableException` because it may be more
> >>> intuitive
> >>>> than `StreamsNotRunningException`.
> >>>>
> >>>> No matter which one picked, it's good to me.
> >>>>
> >>>> --- Vito
> >>>>
> >>>>
> >>>> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax
> >>>> <ma...@confluent.io> wrote:
> >>>>
> >>>>> Thanks for updating the KIP!
> >>>>>
> >>>>> One last comment/question: you kept
> >>>>> `StateStoreNotAvailableException`
> >>> in
> >>>>> favor of `StreamsNotRunningException` (to merge both as
> >>>>> suggested).
> >>>>>
> >>>>> I am wondering, if it might be better to keep
> >>>>> `StreamsNotRunningException` instead of
> >>>>> `StateStoreNotAvailableException`, because this exception
> >>>>> is thrown if Streams is in state PENDING_SHUTDOWN /
> >>>>> NOT_RUNNING / ERROR ?
> >>>>>
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 1/17/20 9:56 PM, John Roesler wrote:
> >>>>>> Thanks, Vito. I've just cast my vote. -John
> >>>>>>
> >>>>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >>>>>>> Hi, folks,
> >>>>>>>
> >>>>>>> Just update the KIP, please take a look.
> >>>>>>>
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>> --- Vito
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng
> >>>>>>> <vi...@is-land.com.tw>
> >>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Bill, John and Matthias. Glad you guys joined
> >>>>>>>> this
> >>> discussion.
> >>>>>>>> I got a lot out of the discussion.
> >>>>>>>>
> >>>>>>>> I would like to update KIP-216 base on John's
> >>>>>>>> suggestion to remove
> >>> the
> >>>>>>>> category.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --- Vito
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
> >>> matthias@confluent.io
> >>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>>> Nevertheless, if we omit the categorization, it’s
> >>>>>>>>>> moot.
> >>>>>>>>>
> >>>>>>>>> Ack.
> >>>>>>>>>
> >>>>>>>>> I am fine to remove the middle tier. As John
> >>>>>>>>> pointed out, it might
> >>> be
> >>>>>>>>> weird to have only one concrete exception type per
> >>>>>>>>> category. We can
> >>>>> also
> >>>>>>>>> explain in detail how to handle each exception in
> >>>>>>>>> their JavaDocs.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >>>>>>>>>> Vito,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the updates, the KIP LGTM.
> >>>>>>>>>>
> >>>>>>>>>> -Bill
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
> >>> vvcephei@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>
> >>>>>>>>>>> Haha, your archive game is on point!
> >>>>>>>>>>>
> >>>>>>>>>>> What Matthias said in that email is essentially
> >>>>>>>>>>> what I figured
> >>> was
> >>>>> the
> >>>>>>>>>>> rationale. It makes sense, but the point I was
> >>>>>>>>>>> making is that
> >>> this
> >>>>>>>>> really
> >>>>>>>>>>> doesn’t seem like a good way to structure a
> >>>>>>>>>>> production app. On
> >>> the
> >>>>>>>>> other
> >>>>>>>>>>> hand, considering the exception fatal has a
> >>>>>>>>>>> good chance of
> >>> avoiding
> >>>>> a
> >>>>>>>>>>> frustrating debug session if you just forgot to
> >>>>>>>>>>> call start.
> >>>>>>>>>>>
> >>>>>>>>>>> Nevertheless, if we omit the categorization,
> >>>>>>>>>>> it’s moot.
> >>>>>>>>>>>
> >>>>>>>>>>> It would be easy to add a categorization layer
> >>>>>>>>>>> later if we want
> >>> it,
> >>>>> but
> >>>>>>>>>>> not very easy to change it if we get it wrong.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for your consideration! -John
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>
> >>>>>>>>>>>> About `StreamsNotStartedException is strange`
> >>>>>>>>>>>> -- The original idea came from Matthias, two
> >>>>>>>>>>>> years ago. :) You can reference here:
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6
> c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> <https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>
> About omitting the categorization --
> >>>>>>>>>>>> It looks reasonable. I'm fine with omitting
> >>>>>>>>>>>> the categorization
> >>> but
> >>>>> not
> >>>>>>>>>>> very
> >>>>>>>>>>>> sure it is a good choice. Does any other
> >>>>>>>>>>>> folks provide opinion?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi, folks,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Just update the KIP-216, please take a look.
> >>>>>>>>>>>>
> >>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng
> >>>>>>>>>>>> <vi...@is-land.com.tw>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi, folks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you suggestion, really appreciate it.
> >>>>>>>>>>>>> :) I understand your concern. I'll merge
> >>> StreamsNotRunningException
> >>>>> and
> >>>>>>>>>>>>> StateStoreNotAvailableException.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John
> >>>>>>>>>>>>> Roesler <
> >>> vvcephei@apache.org
> >>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hey Vito,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes, thanks for the KIP. Sorry the
> >>>>>>>>>>>>>> discussion has been so
> >>> long.
> >>>>>>>>>>>>>> Hopefully, we can close it out soon.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I agree we can drop
> >>>>>>>>>>>>>> StreamsNotRunningException in favor of
> >>>>>>>>>>>>>> just StateStoreNotAvailableException.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Unfortunately, I have some higher-level
> >>>>>>>>>>>>>> concerns. The value of these exceptions
> >>>>>>>>>>>>>> is that they tell you how to handle the
> >>>>>>>>>>>>>> various situations that can arise while
> >>>>>>>>>>>>>> querying a distributed data store.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ideally, as a caller, I should be able to
> >>>>>>>>>>>>>> just catch
> >>> "retriable"
> >>>>> or
> >>>>>>>>>>>>>> "fatal" and handle them appropriately.
> >>>>>>>>>>>>>> Otherwise, there's no point in having
> >>>>>>>>>>>>>> categories, and we should just have all
> >>>>>>>>>>>>>> the exceptions extend
> >>>>>>>>>>>>>> InvalidStateStoreException.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Presently, it's not possible to tell from
> >>>>>>>>>>>>>> just the "retriable"/"fatal" distinction
> >>>>>>>>>>>>>> what to do. You can tell from the
> >>>>>>>>>>>>>> descriptions of the various exceptions.
> >>>>>>>>>>>>>> E.g.:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Retriable: * StreamsRebalancingException:
> >>>>>>>>>>>>>> the exact same call should just be
> >>>>>>>>>>>>>> retried until the rebalance is complete *
> >>>>>>>>>>>>>> StateStoreMigratedException: the store
> >>>>>>>>>>>>>> handle is now invalid, so you need to
> >>>>>>>>>>>>>> re-discover the instance and get a new
> >>>>>>>>>>>>>> handle on that instance. In other words,
> >>>>>>>>>>>>>> the query itself may be valid, but the
> >>>>>>>>>>>>>> particular method invocation on this
> >>>>>>>>>>>>>> particular instance has encountered a
> >>>>>>>>>>>>>> fatal exception.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Fatal: * UnknownStateStoreException: this
> >>>>>>>>>>>>>> is truly fatal. No amount of retrying or
> >>>>>>>>>>>>>> re-discovering is going to get you a
> >>>>>>>>>>>>>> handle
> >>>>> on a
> >>>>>>>>>>>>>> store that doesn't exist in the cluster.
> >>>>>>>>>>>>>> * StateStoreNotAvailableException: this
> >>>>>>>>>>>>>> is actually
> >>> recoverable,
> >>>>>>>>>>>>>> since the store might exist in the
> >>>>>>>>>>>>>> cluster, but isn't
> >>>>> available
> >>>>>>>>> on
> >>>>>>>>>>>>>> this particular instance (which is shut
> >>>>>>>>>>>>>> down or whatever).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Personally, I'm not a fan of code
> >>>>>>>>>>>>>> bureaucracy, so I'm 100%
> >>> fine
> >>>>>>>>>>>>>> with omitting the categorization and just
> >>>>>>>>>>>>>> having 5 subclasses of
> >>>>>>>>>>>>>> InvalidStateStoreException. Each of them
> >>>>>>>>>>>>>> would tell you how to handle them, and
> >>>>>>>>>>>>>> it's not too many to really understand
> >>>>>>>>>>>>>> and handle each one.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> If you really want to have a middle tier,
> >>>>>>>>>>>>>> I'd recommend: *
> >>>>>>>>>>>>>> RetryableStateStoreException: the exact
> >>>>>>>>>>>>>> same call should be repeated. *
> >>>>>>>>>>>>>> RecoverableStateStoreException: the store
> >>>>>>>>>>>>>> handle should be discarded and the caller
> >>>>>>>>>>>>>> should re-discover the location of the
> >>>>>>>>>>>>>> store and repeat the query on the correct
> >>>>>>>>>>>>>> instance. * FatalStateStoreException: the
> >>>>>>>>>>>>>> query/request is totally invalid and will
> >>>>>>>>>>>>>> never succeed.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> However, attempting to categorize the
> >>>>>>>>>>>>>> proposed exceptions reveals even problems
> >>>>>>>>>>>>>> with this categorization: Retriable: *
> >>>>>>>>>>>>>> StreamsRebalancingException Recoverable:
> >>>>>>>>>>>>>> * StateStoreMigratedException *
> >>>>>>>>>>>>>> StreamsNotRunningException Fatal: *
> >>>>>>>>>>>>>> UnknownStateStoreException
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> But StreamsNotStartedException is
> >>>>>>>>>>>>>> strange... It means that one code path
> >>>>>>>>>>>>>> got a handle on a specific KafkaStreams
> >>>>>>>>>>>>>> object instance and sent it a query
> >>>>>>>>>>>>>> before another code path invoked the
> >>>>>>>>>>>>>> start() method on the exact same object
> >>>>>>>>>>>>>> instance. It seems like the most likely
> >>>>>>>>>>>>>> scenario is that whoever wrote the
> >>>>>>>>>>>>>> program just forgot to call start()
> >>>>>>>>>>>>>> before querying, in which case, retrying
> >>>>>>>>>>>>>> isn't going to help, and a fatal
> >>> exception
> >>>>>>>>>>>>>> is more appropriate. I.e., it sounds like
> >>>>>>>>>>>>>> a "first 15 minutes experience" problem,
> >>>>>>>>>>>>>> and making it fatal would be more
> >>>>>>>>>>>>>> helpful. Even in a production context,
> >>>>>>>>>>>>>> there's no reason not to sequence your
> >>>>>>>>>>>>>> application startup such that you don't
> >>>>>>>>>>>>>> accept queries until after Streams is
> >>>>>>>>>>>>>> started. Thus, I guess I'd categorize it
> >>>>>>>>>>>>>> under "fatal".
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regardless of whether you make it fatal
> >>>>>>>>>>>>>> or retriable, you'd still have a whole
> >>>>>>>>>>>>>> category with only one exception in it,
> >>>>>>>>>>>>>> and the other two categories only have
> >>>>>>>>>>>>>> two exceptions. Plus, as you pointed out
> >>>>>>>>>>>>>> in the KIP, you can't get all exceptions
> >>>>>>>>>>>>>> in all cases anyway: * store() can only
> >>>>>>>>>>>>>> throw NotStarted, NotRunning, and
> >>>>>>>>>>>>>> Unknown * actual store queries can only
> >>>>>>>>>>>>>> throw Rebalancing, Migrated, and
> >>>>>>>>>>>>>> NotRunning
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thus, in practice also, there are exactly
> >>>>>>>>>>>>>> three categories and also exactly three
> >>>>>>>>>>>>>> exception types. It doesn't seem like
> >>>>>>>>>>>>>> there's a great advantage to the
> >>>>>>>>>>>>>> categories here. To avoid the
> >>>>>>>>>>>>>> categorization problem and also to
> >>>>>>>>>>>>>> clarify what exceptions can actually be
> >>>>>>>>>>>>>> thrown in different circumstances, it
> >>>>>>>>>>>>>> seems like we should just: * get rid of
> >>>>>>>>>>>>>> the middle tier and make all the
> >>>>>>>>>>>>>> exceptions extend
> >>>>>>>>>>>>>> InvalidStateStoreException * drop
> >>>>>>>>>>>>>> StateStoreNotAvailableException in favor
> >>>>>>>>>>>>>> of StreamsNotRunningException * clearly
> >>>>>>>>>>>>>> document on all public methods which
> >>>>>>>>>>>>>> exceptions need to be handled
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> How do you feel about this? Thanks,
> >>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill
> >>>>>>>>>>>>>> Bejeck wrote:
> >>>>>>>>>>>>>>> Thanks for KIP Vito.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Overall the KIP LGTM, but I'd have to
> >>>>>>>>>>>>>>> agree with others on
> >>>>> merging
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Since in both cases, the thread state
> >>>>>>>>>>>>>>> is in
> >>> `PENDING_SHUTDOWN ||
> >>>>>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure
> >>>>>>>>>>>>>>> how we could
> >>> distinguish
> >>>>>>>>>>> when to
> >>>>>>>>>>>>>>> use the different exceptions. Maybe a
> >>>>>>>>>>>>>>> good middle ground would be to have a
> >>>>> detailed
> >>>>>>>>>>>>>>> exception message.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The KIP freeze is close, so I think if
> >>>>>>>>>>>>>>> we can agree on this,
> >>> we
> >>>>> can
> >>>>>>>>>>>>>> wrap up
> >>>>>>>>>>>>>>> the voting soon.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, Bill
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM
> >>>>>>>>>>>>>>> Matthias J. Sax <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It's still unclear to me what the
> >>>>>>>>>>>>>>>> advantage is, to have both
> >>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For both cased, the state is
> >>>>>>>>>>>>>>>> `PENDING_SHUTDOWN /
> >>> NOT_RUNNING /
> >>>>>>>>>>> ERROR`
> >>>>>>>>>>>>>>>> and thus, for a user point of view,
> >>>>>>>>>>>>>>>> why does it matter if
> >>> the
> >>>>>>>>>>> store is
> >>>>>>>>>>>>>>>> closed on not? I don't understand
> >>>>>>>>>>>>>>>> why/how this information
> >>>>> would
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>> useful? Do you have a concrete
> >>>>>>>>>>>>>>>> example in mind how a user
> >>> would
> >>>>>>>>>>> react
> >>>>>>>>>>>>>>>> differently to both exceptions?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> @Vinoth: about
> >>>>>>>>>>>>>>>> `StreamsRebalancingException` -- to
> >>>>>>>>>>>>>>>> me, it
> >>> seems
> >>>>>>>>>>> best
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> actually do this on a per-query
> >>>>>>>>>>>>>>>> basis, ie, have an overload
> >>>>>>>>>>>>>>>> `KafkaStreams#store(...)` that takes
> >>>>>>>>>>>>>>>> a boolean flag that
> >>> allow
> >>>>> to
> >>>>>>>>>>>>>>>> _disable_ the exception and opt-in to
> >>>>>>>>>>>>>>>> query a active store
> >>>>> during
> >>>>>>>>>>>>>>>> recovery. However, as KIP-535
> >>>>>>>>>>>>>>>> actually introduces this
> >>> change
> >>>>> in
> >>>>>>>>>>>>>>>> behavior, I think KIP-216 should not
> >>>>>>>>>>>>>>>> cover this, but KIP-535
> >>>>>>>>>>> should be
> >>>>>>>>>>>>>>>> updated. I'll follow up on the other
> >>>>>>>>>>>>>>>> KIP thread to raise
> >>> this
> >>>>>>>>>>> point.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> Hi, Matthias & Vinoth,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> What is still unclear to me is,
> >>>>>>>>>>>>>>>>>> what we gain by having
> >>> both
> >>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>> `StateStoreNotAvailableException`. Both
> >>>>>>>>>>>>>>>>>> exception are thrown when
> >>>>>>>>>>>>>>>>>> KafkaStreams is in state
> >>>>>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a
> >>>>>>>>>>>>>>>>>> user what do I gain to
> >>> know
> >>>>>>>>>>> if the
> >>>>>>>>>>>>>>>>>> state store is closed on not -- I
> >>>>>>>>>>>>>>>>>> can't query it anyway?
> >>>>> Maybe
> >>>>>>>>>>> I
> >>>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Yes, both
> >>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>>>>> are fatal exception. But
> >>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
> >>>>>>>>>>>>>>>>> is fatal exception
> >>> about
> >>>>>>>>>>> state
> >>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>> related. I think it would be
> >>>>>>>>>>>>>>>>> helpful that if user need to
> >>> distinguish
> >>>>>>>>>>> these
> >>>>>>>>>>>>>> two
> >>>>>>>>>>>>>>>>> different case to handle it.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'm not very sure, does that make
> >>>>>>>>>>>>>>>>> sense?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM
> >>>>>>>>>>>>>>>>> Vinoth Chandar <
> >>>>>>>>>>> vinoth@apache.org>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> +1 on merging
> >>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`,
> >>>>>>>>>>>>>>>>>> both exceptions are
> >>> fatal
> >>>>>>>>>>>>>> anyway. IMO
> >>>>>>>>>>>>>>>>>> its best to have these exceptions
> >>>>>>>>>>>>>>>>>> be about the state store
> >>>>>>>>>>> (and not
> >>>>>>>>>>>>>>>> streams
> >>>>>>>>>>>>>>>>>> state), to easier understanding.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Additionally, KIP-535 allows for
> >>>>>>>>>>>>>>>>>> querying of state stores
> >>> in
> >>>>>>>>>>>>>> rebalancing
> >>>>>>>>>>>>>>>>>> state. So do we need the
> >>>>>>>>>>>>>>>>>> StreamsRebalancingException?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias
> >>>>>>>>>>>>>>>>>> J. Sax" <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>> Sorry that I dropped the ball
> >>>>>>>>>>>>>>>>>>> on this...
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for updating the KIP.
> >>>>>>>>>>>>>>>>>>> Overall LGTM now. Feel free
> >>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> VOTE
> >>>>>>>>>>>>>>>>>>> thread.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> What is still unclear to me is,
> >>>>>>>>>>>>>>>>>>> what we gain by having
> >>> both
> >>>>>>>>>>>>>>>>>>> `StreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>> `StateStoreNotAvailableException`.
> >>>>>>>>>>>>>>>> Both
> >>>>>>>>>>>>>>>>>>> exception are thrown when
> >>>>>>>>>>>>>>>>>>> KafkaStreams is in state
> >>>>>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as
> >>>>>>>>>>>>>>>>>>> a user what do I gain to
> >>> know
> >>>>>>>>>>> if
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> state store is closed on not --
> >>>>>>>>>>>>>>>>>>> I can't query it anyway?
> >>>>>>>>>>> Maybe I
> >>>>>>>>>>>>>> miss
> >>>>>>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>> Sorry for the late reply,
> >>>>>>>>>>>>>>>>>>>> thanks for the review.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> Why is it only thrown if the state is REBALANCING? A
> >>> store
> >>>>>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>>>> migrated during a
> >>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
> >>>>>>>>>>>>>>>>>>>>> Streams might
> >>>>> resume
> >>>>>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>>>>> RUNNING state and afterward
> >>>>>>>>>>>>>>>>>>>>> somebody tries to use an
> >>> old
> >>>>>>>>>>> store
> >>>>>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>>>>> Also, if state is
> >>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
> >>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
> >>>>>>>>>>>>>>>>>>>>> Hence, I think
> >>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>>>>> does only make sense
> >>> during
> >>>>>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thank you point this, already
> >>>>>>>>>>>>>>>>>>>> updated.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Why do we need to distinguish
> >>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> `KafkaStreamsNotRunningException` may be caused by
> >>> various
> >>>>>>>>>>>>>> reasons, I
> >>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>> it would be helpful that the
> >>>>>>>>>>>>>>>>>>>> user can distinguish whether
> >>>>>>>>>>>>>>>>>>>> it is caused by the state
> >>>>> store
> >>>>>>>>>>>>>> closed.
> >>>>>>>>>>>>>>>>>>>> (Maybe I am wrong...)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Last, why do we distinguish
> >>>>>>>>>>>>>>>>>>>> between `KafkaStreams`
> >>> instance
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it
> >>>>>>>>>>>>>>>>>>>>> seems we should always
> >>> refer to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>>>>> because that is the level
> >>>>>>>>>>>>>>>>>>>>> of granularity in which we
> >>>>>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>>>>> IQ atm.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Totally agree. Do you mean
> >>>>>>>>>>>>>>>>>>>> the naming of state store
> >>>>>>>>>>> exceptions?
> >>>>>>>>>>>>>>>>>>>> I don't have special reason
> >>>>>>>>>>>>>>>>>>>> to distinguish these two.
> >>>>>>>>>>>>>>>>>>>> Your suggestion look more
> >>>>>>>>>>>>>>>>>>>> reasonable for the exception
> >>>>>>>>>>> naming.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Last, for
> >>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
> >>>>>>>>>>>>>>>>>>>> I would add
> >>> that a
> >>>>>>>>>>> user
> >>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> rediscover the store and
> >>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
> >>>>>>>>>>>>>>>>>>>>> the
> >>> store
> >>>>>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>>>>> invalid and a new store
> >>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
> >>>>>>>>>>>>>>>>>>>>> That
> >>> is
> >>>>> a
> >>>>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
> >>>>>>>>>>>>>>>>>>>>> that allows for
> >>>>>>>>>>> "blind"
> >>>>>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>>>>> that either resolve (if the
> >>>>>>>>>>>>>>>>>>>>> store is still on the same
> >>>>>>>>>>> instance
> >>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
> >>>>>>>>>>>>>>>>>>>>> changes to
> >>>>>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> store was migrated away
> >>>>>>>>>>>>>>>>>>>>> during rebalancing).
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The KIP already updated,
> >>>>>>>>>>>>>>>>>>>> please take a look. :)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48
> >>>>>>>>>>>>>>>>>>>> PM Matthias J. Sax <
> >>>>>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Any update on this KIP?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM,
> >>>>>>>>>>>>>>>>>>>>> Matthias J. Sax wrote:
> >>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply.
> >>>>>>>>>>>>>>>>>>>>>> The 2.4 deadline kept us
> >>> quite
> >>>>>>>>>>> busy.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> Why is it only thrown if the state is REBALANCING? A
> >>>>> store
> >>>>>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>>>>> migrated during a
> >>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
> >>>>>>>>>>>>>>>>>>>>>> Streams might
> >>>>> resume
> >>>>>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>>>>>> RUNNING state and
> >>>>>>>>>>>>>>>>>>>>>> afterward somebody tries
> >>>>>>>>>>>>>>>>>>>>>> to use an
> >>> old
> >>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>>>>>> Also, if state is
> >>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
> >>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
> >>>>>>>>>>>>>>>>>>>>>> Hence, I think
> >>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>>>>>> does only make sense
> >>> during
> >>>>>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Why do we need to
> >>>>>>>>>>>>>>>>>>>>>> distinguish between
> >>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Last, why do we
> >>>>>>>>>>>>>>>>>>>>>> distinguish between
> >>>>>>>>>>>>>>>>>>>>>> `KafkaStreams`
> >>>>>>>>>>> instance and
> >>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me,
> >>>>>>>>>>>>>>>>>>>>>> it seems we should
> >>>>>>>>>>>>>>>>>>>>>> always
> >>> refer
> >>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>>>>>> because that is the level
> >>>>>>>>>>>>>>>>>>>>>> of granularity in which
> >>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>>>>> IQ
> >>>>>>>>>>>>>>>>>>>>> atm.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Last, for
> >>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
> >>>>>>>>>>>>>>>>>>>>>> I would add
> >>>>> that a
> >>>>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>> need to
> >>>>>>>>>>>>>>>>>>>>>> rediscover the store and
> >>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
> >>>>>>>>>>>>>>>>>>>>>> the
> >>>>> store
> >>>>>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>>>>>> invalid and a new store
> >>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
> >>>>>>>>>>>>>>>>>>>>>> That
> >>>>> is a
> >>>>>>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
> >>>>>>>>>>>>>>>>>>>>>> that allows for
> >>>>>>>>>>> "blind"
> >>>>>>>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>>>>>> that either resolve (if
> >>>>>>>>>>>>>>>>>>>>>> the store is still on the
> >>>>>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>> instance
> >>>>>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
> >>>>>>>>>>>>>>>>>>>>>> changes to
> >>>>>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> store was migrated away
> >>>>>>>>>>>>>>>>>>>>>> during rebalancing).
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito
> >>>>>>>>>>>>>>>>>>>>>> Jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>> My bad. The short link
> >>>>>>>>>>>>>>>>>>>>>>> `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >>>
> >>>>>>>>>>>>>>>>>>>>>>>
> <https://shorturl.at/CDNT9>
> >>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Please use the
> >>>>>>>>>>>>>>>>>>>>>>> following instead:
> >>>>>>>>>>> https://shorturl.at/bkKQU
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at
> >>>>>>>>>>>>>>>>>>>>>>> 10:53 AM Vito Jeng <
> >>>>>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> Thank you for explanation. I agree with your
> >>> opinion.
> >>>>>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()`
> >>>>>>>>>>>>>>>>>>>>>>>> would never throw
> >>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> For the case that corresponding thread crashes
> >>> after we
> >>>>>>>>>>>>>> handed out
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> store handle. We may
> >>>>>>>>>>>>>>>>>>>>>>>> throw
> >>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>> when stream
> >>> thread is
> >>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>> running(
> >>>>>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9)
> >>>>>>>>>>>>>>>>>>>>>>>> or throw
> >>>>>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>>>>> store is
> >>>>>>>>>>>>>>>>>>>>>>>> closed(https://shorturl.at/hrvAN).
> >>>>>>>>>>>>>>>>>>>>>>>> So I
> >>> think
> >>>>>>>>>>> we
> >>>>>>>>>>>>>> do not
> >>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> add a new type for
> >>>>>>>>>>>>>>>>>>>>>>>> this case. Does that
> >>>>>>>>>>>>>>>>>>>>>>>> make sense?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> About
> >>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>>>>>>>> vs
> >>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> I understand your point. I rename
> >>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> About check unknown state store names:
> >>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
> >>>>>>>>>>>>>>>>>>>>>>>> hint. I add a new
> >>>>>>>>>>>>>>>>>>>>>>>> type
> >>>>>>>>>>>>>>>>>> `UnknownStateStoreException`
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>> this case.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Also, we should
> >>>>>>>>>>>>>>>>>>>>>>>>> still have fatal
> >>>>>>>>>>>>>>>>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>>>>>> Not sure why you
> >>>>>>>>>>> remove
> >>>>>>>>>>>>>> it?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thank you point this,
> >>>>>>>>>>>>>>>>>>>>>>>> already add it
> >>>>>>>>>>>>>>>>>>>>>>>> again.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The KIP already
> >>>>>>>>>>>>>>>>>>>>>>>> updated, please take
> >>>>>>>>>>>>>>>>>>>>>>>> a look.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> --- Vito
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
> -----BEGIN PGP SIGNATURE-----
>
> iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl45vCwACgkQO4miYXKq
> /OhT4Q//elvNj2BRIMd7SqyQhpa6sSq4k2iG3wFlr/B2xkmkZPWUICt3SpCisIxG
> RlP5ml7Mi8IcWv9jmMux9C1NpftboLudxIUcun/I4cP0T3S7ytXfzdO+V1WuD9xe
> hfeqMwJTytusEv5VO3MyY+jYGqmPJrPJRViZ2Uwsj3Ojk0SZRb/m+b6ah6BGMh0E
> nlUIux7cabg1OZ/ee0x21hcAB32lVIRfPLKZeHCTHuYmlW76X4pXhFxOm69Pumtj
> ciVMr/pl7B+XiMk0C2Po04zQcP8+5/O7LYU4e8ha12NtuNuDTlQLGUb+S9Qrbxb2
> xdFrlC81RROmIdIOTLiJeeBqhzd19llDgMhBf4spJMOeLn7359PX6r/9tDqonJrQ
> wpSp4S54MDYfTPHUPqA6u/FsE0BX4EMU71ckC9rXVRRUnQ8A8cMk/6qWlRu8NvxQ
> IiYuSo48UENlcTHxXC/rzIDp54gvtv2iml8QcOg6tS+hzhAI4yoY+1w4pnjPmRo0
> 4D1hnq5bd1SLrGSGcCxVbN0jtwgkcr50HBb1UkIDRndqapfKwZOMV65tIwTvxaaZ
> r2QPYyPd6ZDgeMdy0r94z0SMwatqGCmJD9EQmSmLBulemrvYxPGUevQls+WgE4WE
> zc3LtoyfdlXlkttTU0Q/dm2H9OU4s3QO+6PEdC+Qwi3aCKKhdUQ=
> =4EMC
> -----END PGP SIGNATURE-----
>
>
>
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Navinder Brar <na...@yahoo.com.INVALID>.
Thanks Vito, for incorporating this. Makes sense.
-Navinder
On Wednesday, February 5, 2020, 12:17 AM, Matthias J. Sax <mj...@apache.org> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Thanks Vito!
That makes sense to me.
On 2/1/20 11:29 PM, Vito Jeng wrote:
> Hi, folks,
>
> KIP-562(KAFKA-9445) already merged three days ago.
>
> I have updated KIP-216 to reflect the KIP-562. The main change is
> to introduce a new exception `InvalidStateStorePartitionException`,
> will be thrown when user requested partition not available.
>
> Please take a look and any feedback is welcome. Thanks Matthias for
> the reminder.
>
> --- Vito
>
>
> On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng <vi...@is-land.com.tw>
> wrote:
>
>> Got it, thanks Matthias.
>>
>> --- Vito
>>
>>
>> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
>> <ma...@confluent.io> wrote:
>>
>>> Thanks Vito.
>>>
>>> I am also ok with either name. Just a personal slight
>>> preference, but not a important.
>>>
>>>
>>> -Matthias
>>>
>>> On 1/21/20 6:52 PM, Vito Jeng wrote:
>>>> Thanks Matthias.
>>>>
>>>> The KIP is about InvalidStateStoreException. I pick
>>>> `StateStoreNotAvailableException` because it may be more
>>> intuitive
>>>> than `StreamsNotRunningException`.
>>>>
>>>> No matter which one picked, it's good to me.
>>>>
>>>> --- Vito
>>>>
>>>>
>>>> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax
>>>> <ma...@confluent.io> wrote:
>>>>
>>>>> Thanks for updating the KIP!
>>>>>
>>>>> One last comment/question: you kept
>>>>> `StateStoreNotAvailableException`
>>> in
>>>>> favor of `StreamsNotRunningException` (to merge both as
>>>>> suggested).
>>>>>
>>>>> I am wondering, if it might be better to keep
>>>>> `StreamsNotRunningException` instead of
>>>>> `StateStoreNotAvailableException`, because this exception
>>>>> is thrown if Streams is in state PENDING_SHUTDOWN /
>>>>> NOT_RUNNING / ERROR ?
>>>>>
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 1/17/20 9:56 PM, John Roesler wrote:
>>>>>> Thanks, Vito. I've just cast my vote. -John
>>>>>>
>>>>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>>>>>>> Hi, folks,
>>>>>>>
>>>>>>> Just update the KIP, please take a look.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --- Vito
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng
>>>>>>> <vi...@is-land.com.tw>
>>> wrote:
>>>>>>>
>>>>>>>> Thanks Bill, John and Matthias. Glad you guys joined
>>>>>>>> this
>>> discussion.
>>>>>>>> I got a lot out of the discussion.
>>>>>>>>
>>>>>>>> I would like to update KIP-216 base on John's
>>>>>>>> suggestion to remove
>>> the
>>>>>>>> category.
>>>>>>>>
>>>>>>>>
>>>>>>>> --- Vito
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
>>> matthias@confluent.io
>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>> Nevertheless, if we omit the categorization, it’s
>>>>>>>>>> moot.
>>>>>>>>>
>>>>>>>>> Ack.
>>>>>>>>>
>>>>>>>>> I am fine to remove the middle tier. As John
>>>>>>>>> pointed out, it might
>>> be
>>>>>>>>> weird to have only one concrete exception type per
>>>>>>>>> category. We can
>>>>> also
>>>>>>>>> explain in detail how to handle each exception in
>>>>>>>>> their JavaDocs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>>>>>>>>>> Vito,
>>>>>>>>>>
>>>>>>>>>> Thanks for the updates, the KIP LGTM.
>>>>>>>>>>
>>>>>>>>>> -Bill
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
>>> vvcephei@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>
>>>>>>>>>>> Haha, your archive game is on point!
>>>>>>>>>>>
>>>>>>>>>>> What Matthias said in that email is essentially
>>>>>>>>>>> what I figured
>>> was
>>>>> the
>>>>>>>>>>> rationale. It makes sense, but the point I was
>>>>>>>>>>> making is that
>>> this
>>>>>>>>> really
>>>>>>>>>>> doesn’t seem like a good way to structure a
>>>>>>>>>>> production app. On
>>> the
>>>>>>>>> other
>>>>>>>>>>> hand, considering the exception fatal has a
>>>>>>>>>>> good chance of
>>> avoiding
>>>>> a
>>>>>>>>>>> frustrating debug session if you just forgot to
>>>>>>>>>>> call start.
>>>>>>>>>>>
>>>>>>>>>>> Nevertheless, if we omit the categorization,
>>>>>>>>>>> it’s moot.
>>>>>>>>>>>
>>>>>>>>>>> It would be easy to add a categorization layer
>>>>>>>>>>> later if we want
>>> it,
>>>>> but
>>>>>>>>>>> not very easy to change it if we get it wrong.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your consideration! -John
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>
>>>>>>>>>>>> About `StreamsNotStartedException is strange`
>>>>>>>>>>>> -- The original idea came from Matthias, two
>>>>>>>>>>>> years ago. :) You can reference here:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>
>>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6
c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>
About omitting the categorization --
>>>>>>>>>>>> It looks reasonable. I'm fine with omitting
>>>>>>>>>>>> the categorization
>>> but
>>>>> not
>>>>>>>>>>> very
>>>>>>>>>>>> sure it is a good choice. Does any other
>>>>>>>>>>>> folks provide opinion?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi, folks,
>>>>>>>>>>>>
>>>>>>>>>>>> Just update the KIP-216, please take a look.
>>>>>>>>>>>>
>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng
>>>>>>>>>>>> <vi...@is-land.com.tw>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, folks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you suggestion, really appreciate it.
>>>>>>>>>>>>> :) I understand your concern. I'll merge
>>> StreamsNotRunningException
>>>>> and
>>>>>>>>>>>>> StateStoreNotAvailableException.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John
>>>>>>>>>>>>> Roesler <
>>> vvcephei@apache.org
>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Vito,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, thanks for the KIP. Sorry the
>>>>>>>>>>>>>> discussion has been so
>>> long.
>>>>>>>>>>>>>> Hopefully, we can close it out soon.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree we can drop
>>>>>>>>>>>>>> StreamsNotRunningException in favor of
>>>>>>>>>>>>>> just StateStoreNotAvailableException.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unfortunately, I have some higher-level
>>>>>>>>>>>>>> concerns. The value of these exceptions
>>>>>>>>>>>>>> is that they tell you how to handle the
>>>>>>>>>>>>>> various situations that can arise while
>>>>>>>>>>>>>> querying a distributed data store.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ideally, as a caller, I should be able to
>>>>>>>>>>>>>> just catch
>>> "retriable"
>>>>> or
>>>>>>>>>>>>>> "fatal" and handle them appropriately.
>>>>>>>>>>>>>> Otherwise, there's no point in having
>>>>>>>>>>>>>> categories, and we should just have all
>>>>>>>>>>>>>> the exceptions extend
>>>>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Presently, it's not possible to tell from
>>>>>>>>>>>>>> just the "retriable"/"fatal" distinction
>>>>>>>>>>>>>> what to do. You can tell from the
>>>>>>>>>>>>>> descriptions of the various exceptions.
>>>>>>>>>>>>>> E.g.:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Retriable: * StreamsRebalancingException:
>>>>>>>>>>>>>> the exact same call should just be
>>>>>>>>>>>>>> retried until the rebalance is complete *
>>>>>>>>>>>>>> StateStoreMigratedException: the store
>>>>>>>>>>>>>> handle is now invalid, so you need to
>>>>>>>>>>>>>> re-discover the instance and get a new
>>>>>>>>>>>>>> handle on that instance. In other words,
>>>>>>>>>>>>>> the query itself may be valid, but the
>>>>>>>>>>>>>> particular method invocation on this
>>>>>>>>>>>>>> particular instance has encountered a
>>>>>>>>>>>>>> fatal exception.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fatal: * UnknownStateStoreException: this
>>>>>>>>>>>>>> is truly fatal. No amount of retrying or
>>>>>>>>>>>>>> re-discovering is going to get you a
>>>>>>>>>>>>>> handle
>>>>> on a
>>>>>>>>>>>>>> store that doesn't exist in the cluster.
>>>>>>>>>>>>>> * StateStoreNotAvailableException: this
>>>>>>>>>>>>>> is actually
>>> recoverable,
>>>>>>>>>>>>>> since the store might exist in the
>>>>>>>>>>>>>> cluster, but isn't
>>>>> available
>>>>>>>>> on
>>>>>>>>>>>>>> this particular instance (which is shut
>>>>>>>>>>>>>> down or whatever).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Personally, I'm not a fan of code
>>>>>>>>>>>>>> bureaucracy, so I'm 100%
>>> fine
>>>>>>>>>>>>>> with omitting the categorization and just
>>>>>>>>>>>>>> having 5 subclasses of
>>>>>>>>>>>>>> InvalidStateStoreException. Each of them
>>>>>>>>>>>>>> would tell you how to handle them, and
>>>>>>>>>>>>>> it's not too many to really understand
>>>>>>>>>>>>>> and handle each one.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you really want to have a middle tier,
>>>>>>>>>>>>>> I'd recommend: *
>>>>>>>>>>>>>> RetryableStateStoreException: the exact
>>>>>>>>>>>>>> same call should be repeated. *
>>>>>>>>>>>>>> RecoverableStateStoreException: the store
>>>>>>>>>>>>>> handle should be discarded and the caller
>>>>>>>>>>>>>> should re-discover the location of the
>>>>>>>>>>>>>> store and repeat the query on the correct
>>>>>>>>>>>>>> instance. * FatalStateStoreException: the
>>>>>>>>>>>>>> query/request is totally invalid and will
>>>>>>>>>>>>>> never succeed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, attempting to categorize the
>>>>>>>>>>>>>> proposed exceptions reveals even problems
>>>>>>>>>>>>>> with this categorization: Retriable: *
>>>>>>>>>>>>>> StreamsRebalancingException Recoverable:
>>>>>>>>>>>>>> * StateStoreMigratedException *
>>>>>>>>>>>>>> StreamsNotRunningException Fatal: *
>>>>>>>>>>>>>> UnknownStateStoreException
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But StreamsNotStartedException is
>>>>>>>>>>>>>> strange... It means that one code path
>>>>>>>>>>>>>> got a handle on a specific KafkaStreams
>>>>>>>>>>>>>> object instance and sent it a query
>>>>>>>>>>>>>> before another code path invoked the
>>>>>>>>>>>>>> start() method on the exact same object
>>>>>>>>>>>>>> instance. It seems like the most likely
>>>>>>>>>>>>>> scenario is that whoever wrote the
>>>>>>>>>>>>>> program just forgot to call start()
>>>>>>>>>>>>>> before querying, in which case, retrying
>>>>>>>>>>>>>> isn't going to help, and a fatal
>>> exception
>>>>>>>>>>>>>> is more appropriate. I.e., it sounds like
>>>>>>>>>>>>>> a "first 15 minutes experience" problem,
>>>>>>>>>>>>>> and making it fatal would be more
>>>>>>>>>>>>>> helpful. Even in a production context,
>>>>>>>>>>>>>> there's no reason not to sequence your
>>>>>>>>>>>>>> application startup such that you don't
>>>>>>>>>>>>>> accept queries until after Streams is
>>>>>>>>>>>>>> started. Thus, I guess I'd categorize it
>>>>>>>>>>>>>> under "fatal".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regardless of whether you make it fatal
>>>>>>>>>>>>>> or retriable, you'd still have a whole
>>>>>>>>>>>>>> category with only one exception in it,
>>>>>>>>>>>>>> and the other two categories only have
>>>>>>>>>>>>>> two exceptions. Plus, as you pointed out
>>>>>>>>>>>>>> in the KIP, you can't get all exceptions
>>>>>>>>>>>>>> in all cases anyway: * store() can only
>>>>>>>>>>>>>> throw NotStarted, NotRunning, and
>>>>>>>>>>>>>> Unknown * actual store queries can only
>>>>>>>>>>>>>> throw Rebalancing, Migrated, and
>>>>>>>>>>>>>> NotRunning
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thus, in practice also, there are exactly
>>>>>>>>>>>>>> three categories and also exactly three
>>>>>>>>>>>>>> exception types. It doesn't seem like
>>>>>>>>>>>>>> there's a great advantage to the
>>>>>>>>>>>>>> categories here. To avoid the
>>>>>>>>>>>>>> categorization problem and also to
>>>>>>>>>>>>>> clarify what exceptions can actually be
>>>>>>>>>>>>>> thrown in different circumstances, it
>>>>>>>>>>>>>> seems like we should just: * get rid of
>>>>>>>>>>>>>> the middle tier and make all the
>>>>>>>>>>>>>> exceptions extend
>>>>>>>>>>>>>> InvalidStateStoreException * drop
>>>>>>>>>>>>>> StateStoreNotAvailableException in favor
>>>>>>>>>>>>>> of StreamsNotRunningException * clearly
>>>>>>>>>>>>>> document on all public methods which
>>>>>>>>>>>>>> exceptions need to be handled
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How do you feel about this? Thanks,
>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill
>>>>>>>>>>>>>> Bejeck wrote:
>>>>>>>>>>>>>>> Thanks for KIP Vito.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Overall the KIP LGTM, but I'd have to
>>>>>>>>>>>>>>> agree with others on
>>>>> merging
>>>>>>>>>>> the
>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Since in both cases, the thread state
>>>>>>>>>>>>>>> is in
>>> `PENDING_SHUTDOWN ||
>>>>>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure
>>>>>>>>>>>>>>> how we could
>>> distinguish
>>>>>>>>>>> when to
>>>>>>>>>>>>>>> use the different exceptions. Maybe a
>>>>>>>>>>>>>>> good middle ground would be to have a
>>>>> detailed
>>>>>>>>>>>>>>> exception message.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The KIP freeze is close, so I think if
>>>>>>>>>>>>>>> we can agree on this,
>>> we
>>>>> can
>>>>>>>>>>>>>> wrap up
>>>>>>>>>>>>>>> the voting soon.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks, Bill
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM
>>>>>>>>>>>>>>> Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's still unclear to me what the
>>>>>>>>>>>>>>>> advantage is, to have both
>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For both cased, the state is
>>>>>>>>>>>>>>>> `PENDING_SHUTDOWN /
>>> NOT_RUNNING /
>>>>>>>>>>> ERROR`
>>>>>>>>>>>>>>>> and thus, for a user point of view,
>>>>>>>>>>>>>>>> why does it matter if
>>> the
>>>>>>>>>>> store is
>>>>>>>>>>>>>>>> closed on not? I don't understand
>>>>>>>>>>>>>>>> why/how this information
>>>>> would
>>>>>>>>>>> be
>>>>>>>>>>>>>>>> useful? Do you have a concrete
>>>>>>>>>>>>>>>> example in mind how a user
>>> would
>>>>>>>>>>> react
>>>>>>>>>>>>>>>> differently to both exceptions?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Vinoth: about
>>>>>>>>>>>>>>>> `StreamsRebalancingException` -- to
>>>>>>>>>>>>>>>> me, it
>>> seems
>>>>>>>>>>> best
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> actually do this on a per-query
>>>>>>>>>>>>>>>> basis, ie, have an overload
>>>>>>>>>>>>>>>> `KafkaStreams#store(...)` that takes
>>>>>>>>>>>>>>>> a boolean flag that
>>> allow
>>>>> to
>>>>>>>>>>>>>>>> _disable_ the exception and opt-in to
>>>>>>>>>>>>>>>> query a active store
>>>>> during
>>>>>>>>>>>>>>>> recovery. However, as KIP-535
>>>>>>>>>>>>>>>> actually introduces this
>>> change
>>>>> in
>>>>>>>>>>>>>>>> behavior, I think KIP-216 should not
>>>>>>>>>>>>>>>> cover this, but KIP-535
>>>>>>>>>>> should be
>>>>>>>>>>>>>>>> updated. I'll follow up on the other
>>>>>>>>>>>>>>>> KIP thread to raise
>>> this
>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Hi, Matthias & Vinoth,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What is still unclear to me is,
>>>>>>>>>>>>>>>>>> what we gain by having
>>> both
>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>> `StateStoreNotAvailableException`. Both
>>>>>>>>>>>>>>>>>> exception are thrown when
>>>>>>>>>>>>>>>>>> KafkaStreams is in state
>>>>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a
>>>>>>>>>>>>>>>>>> user what do I gain to
>>> know
>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>> state store is closed on not -- I
>>>>>>>>>>>>>>>>>> can't query it anyway?
>>>>> Maybe
>>>>>>>>>>> I
>>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, both
>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>>>>> are fatal exception. But
>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>>>>> is fatal exception
>>> about
>>>>>>>>>>> state
>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>> related. I think it would be
>>>>>>>>>>>>>>>>> helpful that if user need to
>>> distinguish
>>>>>>>>>>> these
>>>>>>>>>>>>>> two
>>>>>>>>>>>>>>>>> different case to handle it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm not very sure, does that make
>>>>>>>>>>>>>>>>> sense?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM
>>>>>>>>>>>>>>>>> Vinoth Chandar <
>>>>>>>>>>> vinoth@apache.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +1 on merging
>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`,
>>>>>>>>>>>>>>>>>> both exceptions are
>>> fatal
>>>>>>>>>>>>>> anyway. IMO
>>>>>>>>>>>>>>>>>> its best to have these exceptions
>>>>>>>>>>>>>>>>>> be about the state store
>>>>>>>>>>> (and not
>>>>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>>>>> state), to easier understanding.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Additionally, KIP-535 allows for
>>>>>>>>>>>>>>>>>> querying of state stores
>>> in
>>>>>>>>>>>>>> rebalancing
>>>>>>>>>>>>>>>>>> state. So do we need the
>>>>>>>>>>>>>>>>>> StreamsRebalancingException?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias
>>>>>>>>>>>>>>>>>> J. Sax" <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Sorry that I dropped the ball
>>>>>>>>>>>>>>>>>>> on this...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP.
>>>>>>>>>>>>>>>>>>> Overall LGTM now. Feel free
>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> VOTE
>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What is still unclear to me is,
>>>>>>>>>>>>>>>>>>> what we gain by having
>>> both
>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException`
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> `StateStoreNotAvailableException`.
>>>>>>>>>>>>>>>> Both
>>>>>>>>>>>>>>>>>>> exception are thrown when
>>>>>>>>>>>>>>>>>>> KafkaStreams is in state
>>>>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as
>>>>>>>>>>>>>>>>>>> a user what do I gain to
>>> know
>>>>>>>>>>> if
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> state store is closed on not --
>>>>>>>>>>>>>>>>>>> I can't query it anyway?
>>>>>>>>>>> Maybe I
>>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Sorry for the late reply,
>>>>>>>>>>>>>>>>>>>> thanks for the review.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
Why is it only thrown if the state is REBALANCING? A
>>> store
>>>>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>>> migrated during a
>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
>>>>>>>>>>>>>>>>>>>>> Streams might
>>>>> resume
>>>>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>>>> RUNNING state and afterward
>>>>>>>>>>>>>>>>>>>>> somebody tries to use an
>>> old
>>>>>>>>>>> store
>>>>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>>>>> Also, if state is
>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
>>>>>>>>>>>>>>>>>>>>> Hence, I think
>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>>>>> does only make sense
>>> during
>>>>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you point this, already
>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Why do we need to distinguish
>>>>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
`KafkaStreamsNotRunningException` may be caused by
>>> various
>>>>>>>>>>>>>> reasons, I
>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>> it would be helpful that the
>>>>>>>>>>>>>>>>>>>> user can distinguish whether
>>>>>>>>>>>>>>>>>>>> it is caused by the state
>>>>> store
>>>>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>>>>> (Maybe I am wrong...)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Last, why do we distinguish
>>>>>>>>>>>>>>>>>>>> between `KafkaStreams`
>>> instance
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it
>>>>>>>>>>>>>>>>>>>>> seems we should always
>>> refer to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>>>>> because that is the level
>>>>>>>>>>>>>>>>>>>>> of granularity in which we
>>>>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>>>>> IQ atm.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Totally agree. Do you mean
>>>>>>>>>>>>>>>>>>>> the naming of state store
>>>>>>>>>>> exceptions?
>>>>>>>>>>>>>>>>>>>> I don't have special reason
>>>>>>>>>>>>>>>>>>>> to distinguish these two.
>>>>>>>>>>>>>>>>>>>> Your suggestion look more
>>>>>>>>>>>>>>>>>>>> reasonable for the exception
>>>>>>>>>>> naming.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Last, for
>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
>>>>>>>>>>>>>>>>>>>> I would add
>>> that a
>>>>>>>>>>> user
>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> rediscover the store and
>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
>>>>>>>>>>>>>>>>>>>>> the
>>> store
>>>>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>>>>> invalid and a new store
>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
>>>>>>>>>>>>>>>>>>>>> That
>>> is
>>>>> a
>>>>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
>>>>>>>>>>>>>>>>>>>>> that allows for
>>>>>>>>>>> "blind"
>>>>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>>>>> that either resolve (if the
>>>>>>>>>>>>>>>>>>>>> store is still on the same
>>>>>>>>>>> instance
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
>>>>>>>>>>>>>>>>>>>>> changes to
>>>>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> store was migrated away
>>>>>>>>>>>>>>>>>>>>> during rebalancing).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The KIP already updated,
>>>>>>>>>>>>>>>>>>>> please take a look. :)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48
>>>>>>>>>>>>>>>>>>>> PM Matthias J. Sax <
>>>>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Any update on this KIP?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM,
>>>>>>>>>>>>>>>>>>>>> Matthias J. Sax wrote:
>>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply.
>>>>>>>>>>>>>>>>>>>>>> The 2.4 deadline kept us
>>> quite
>>>>>>>>>>> busy.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
Why is it only thrown if the state is REBALANCING? A
>>>>> store
>>>>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>>>> migrated during a
>>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
>>>>>>>>>>>>>>>>>>>>>> Streams might
>>>>> resume
>>>>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>>>>> RUNNING state and
>>>>>>>>>>>>>>>>>>>>>> afterward somebody tries
>>>>>>>>>>>>>>>>>>>>>> to use an
>>> old
>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>>>>>> Also, if state is
>>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
>>>>>>>>>>>>>>>>>>>>>> Hence, I think
>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>>>>>> does only make sense
>>> during
>>>>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Why do we need to
>>>>>>>>>>>>>>>>>>>>>> distinguish between
>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Last, why do we
>>>>>>>>>>>>>>>>>>>>>> distinguish between
>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams`
>>>>>>>>>>> instance and
>>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me,
>>>>>>>>>>>>>>>>>>>>>> it seems we should
>>>>>>>>>>>>>>>>>>>>>> always
>>> refer
>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>>>>>> because that is the level
>>>>>>>>>>>>>>>>>>>>>> of granularity in which
>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>>>>> IQ
>>>>>>>>>>>>>>>>>>>>> atm.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Last, for
>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
>>>>>>>>>>>>>>>>>>>>>> I would add
>>>>> that a
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>> rediscover the store and
>>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
>>>>>>>>>>>>>>>>>>>>>> the
>>>>> store
>>>>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>>>>>> invalid and a new store
>>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
>>>>>>>>>>>>>>>>>>>>>> That
>>>>> is a
>>>>>>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
>>>>>>>>>>>>>>>>>>>>>> that allows for
>>>>>>>>>>> "blind"
>>>>>>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>>>>>> that either resolve (if
>>>>>>>>>>>>>>>>>>>>>> the store is still on the
>>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>> instance
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
>>>>>>>>>>>>>>>>>>>>>> changes to
>>>>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> store was migrated away
>>>>>>>>>>>>>>>>>>>>>> during rebalancing).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito
>>>>>>>>>>>>>>>>>>>>>> Jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>> My bad. The short link
>>>>>>>>>>>>>>>>>>>>>>> `https://shorturl.at/CDNT9`
>>>
>>>>>>>>>>>>>>>>>>>>>>>
<https://shorturl.at/CDNT9>
>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
<https://shorturl.at/CDNT9> seems incorrect.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Please use the
>>>>>>>>>>>>>>>>>>>>>>> following instead:
>>>>>>>>>>> https://shorturl.at/bkKQU
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at
>>>>>>>>>>>>>>>>>>>>>>> 10:53 AM Vito Jeng <
>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
Thank you for explanation. I agree with your
>>> opinion.
>>>>>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()`
>>>>>>>>>>>>>>>>>>>>>>>> would never throw
>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
For the case that corresponding thread crashes
>>> after we
>>>>>>>>>>>>>> handed out
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> store handle. We may
>>>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
In `StreamThreadStateStoreProvider`, we would throw
>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>> when stream
>>> thread is
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> running(
>>>>>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9)
>>>>>>>>>>>>>>>>>>>>>>>> or throw
>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>> store is
>>>>>>>>>>>>>>>>>>>>>>>> closed(https://shorturl.at/hrvAN).
>>>>>>>>>>>>>>>>>>>>>>>> So I
>>> think
>>>>>>>>>>> we
>>>>>>>>>>>>>> do not
>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> add a new type for
>>>>>>>>>>>>>>>>>>>>>>>> this case. Does that
>>>>>>>>>>>>>>>>>>>>>>>> make sense?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>>> vs
>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
I understand your point. I rename
>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
About check unknown state store names:
>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
>>>>>>>>>>>>>>>>>>>>>>>> hint. I add a new
>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>> `UnknownStateStoreException`
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>> this case.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Also, we should
>>>>>>>>>>>>>>>>>>>>>>>>> still have fatal
>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>>>> Not sure why you
>>>>>>>>>>> remove
>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thank you point this,
>>>>>>>>>>>>>>>>>>>>>>>> already add it
>>>>>>>>>>>>>>>>>>>>>>>> again.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The KIP already
>>>>>>>>>>>>>>>>>>>>>>>> updated, please take
>>>>>>>>>>>>>>>>>>>>>>>> a look.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl45vCwACgkQO4miYXKq
/OhT4Q//elvNj2BRIMd7SqyQhpa6sSq4k2iG3wFlr/B2xkmkZPWUICt3SpCisIxG
RlP5ml7Mi8IcWv9jmMux9C1NpftboLudxIUcun/I4cP0T3S7ytXfzdO+V1WuD9xe
hfeqMwJTytusEv5VO3MyY+jYGqmPJrPJRViZ2Uwsj3Ojk0SZRb/m+b6ah6BGMh0E
nlUIux7cabg1OZ/ee0x21hcAB32lVIRfPLKZeHCTHuYmlW76X4pXhFxOm69Pumtj
ciVMr/pl7B+XiMk0C2Po04zQcP8+5/O7LYU4e8ha12NtuNuDTlQLGUb+S9Qrbxb2
xdFrlC81RROmIdIOTLiJeeBqhzd19llDgMhBf4spJMOeLn7359PX6r/9tDqonJrQ
wpSp4S54MDYfTPHUPqA6u/FsE0BX4EMU71ckC9rXVRRUnQ8A8cMk/6qWlRu8NvxQ
IiYuSo48UENlcTHxXC/rzIDp54gvtv2iml8QcOg6tS+hzhAI4yoY+1w4pnjPmRo0
4D1hnq5bd1SLrGSGcCxVbN0jtwgkcr50HBb1UkIDRndqapfKwZOMV65tIwTvxaaZ
r2QPYyPd6ZDgeMdy0r94z0SMwatqGCmJD9EQmSmLBulemrvYxPGUevQls+WgE4WE
zc3LtoyfdlXlkttTU0Q/dm2H9OU4s3QO+6PEdC+Qwi3aCKKhdUQ=
=4EMC
-----END PGP SIGNATURE-----
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <mj...@apache.org>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Thanks Vito!
That makes sense to me.
On 2/1/20 11:29 PM, Vito Jeng wrote:
> Hi, folks,
>
> KIP-562(KAFKA-9445) already merged three days ago.
>
> I have updated KIP-216 to reflect the KIP-562. The main change is
> to introduce a new exception `InvalidStateStorePartitionException`,
> will be thrown when user requested partition not available.
>
> Please take a look and any feedback is welcome. Thanks Matthias for
> the reminder.
>
> --- Vito
>
>
> On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng <vi...@is-land.com.tw>
> wrote:
>
>> Got it, thanks Matthias.
>>
>> --- Vito
>>
>>
>> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax
>> <ma...@confluent.io> wrote:
>>
>>> Thanks Vito.
>>>
>>> I am also ok with either name. Just a personal slight
>>> preference, but not a important.
>>>
>>>
>>> -Matthias
>>>
>>> On 1/21/20 6:52 PM, Vito Jeng wrote:
>>>> Thanks Matthias.
>>>>
>>>> The KIP is about InvalidStateStoreException. I pick
>>>> `StateStoreNotAvailableException` because it may be more
>>> intuitive
>>>> than `StreamsNotRunningException`.
>>>>
>>>> No matter which one picked, it's good to me.
>>>>
>>>> --- Vito
>>>>
>>>>
>>>> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax
>>>> <ma...@confluent.io> wrote:
>>>>
>>>>> Thanks for updating the KIP!
>>>>>
>>>>> One last comment/question: you kept
>>>>> `StateStoreNotAvailableException`
>>> in
>>>>> favor of `StreamsNotRunningException` (to merge both as
>>>>> suggested).
>>>>>
>>>>> I am wondering, if it might be better to keep
>>>>> `StreamsNotRunningException` instead of
>>>>> `StateStoreNotAvailableException`, because this exception
>>>>> is thrown if Streams is in state PENDING_SHUTDOWN /
>>>>> NOT_RUNNING / ERROR ?
>>>>>
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 1/17/20 9:56 PM, John Roesler wrote:
>>>>>> Thanks, Vito. I've just cast my vote. -John
>>>>>>
>>>>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>>>>>>> Hi, folks,
>>>>>>>
>>>>>>> Just update the KIP, please take a look.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> --- Vito
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng
>>>>>>> <vi...@is-land.com.tw>
>>> wrote:
>>>>>>>
>>>>>>>> Thanks Bill, John and Matthias. Glad you guys joined
>>>>>>>> this
>>> discussion.
>>>>>>>> I got a lot out of the discussion.
>>>>>>>>
>>>>>>>> I would like to update KIP-216 base on John's
>>>>>>>> suggestion to remove
>>> the
>>>>>>>> category.
>>>>>>>>
>>>>>>>>
>>>>>>>> --- Vito
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
>>> matthias@confluent.io
>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>> Nevertheless, if we omit the categorization, it’s
>>>>>>>>>> moot.
>>>>>>>>>
>>>>>>>>> Ack.
>>>>>>>>>
>>>>>>>>> I am fine to remove the middle tier. As John
>>>>>>>>> pointed out, it might
>>> be
>>>>>>>>> weird to have only one concrete exception type per
>>>>>>>>> category. We can
>>>>> also
>>>>>>>>> explain in detail how to handle each exception in
>>>>>>>>> their JavaDocs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>>>>>>>>>> Vito,
>>>>>>>>>>
>>>>>>>>>> Thanks for the updates, the KIP LGTM.
>>>>>>>>>>
>>>>>>>>>> -Bill
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
>>> vvcephei@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>
>>>>>>>>>>> Haha, your archive game is on point!
>>>>>>>>>>>
>>>>>>>>>>> What Matthias said in that email is essentially
>>>>>>>>>>> what I figured
>>> was
>>>>> the
>>>>>>>>>>> rationale. It makes sense, but the point I was
>>>>>>>>>>> making is that
>>> this
>>>>>>>>> really
>>>>>>>>>>> doesn’t seem like a good way to structure a
>>>>>>>>>>> production app. On
>>> the
>>>>>>>>> other
>>>>>>>>>>> hand, considering the exception fatal has a
>>>>>>>>>>> good chance of
>>> avoiding
>>>>> a
>>>>>>>>>>> frustrating debug session if you just forgot to
>>>>>>>>>>> call start.
>>>>>>>>>>>
>>>>>>>>>>> Nevertheless, if we omit the categorization,
>>>>>>>>>>> it’s moot.
>>>>>>>>>>>
>>>>>>>>>>> It would be easy to add a categorization layer
>>>>>>>>>>> later if we want
>>> it,
>>>>> but
>>>>>>>>>>> not very easy to change it if we get it wrong.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your consideration! -John
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>
>>>>>>>>>>>> About `StreamsNotStartedException is strange`
>>>>>>>>>>>> -- The original idea came from Matthias, two
>>>>>>>>>>>> years ago. :) You can reference here:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>
>>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6
c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>
About omitting the categorization --
>>>>>>>>>>>> It looks reasonable. I'm fine with omitting
>>>>>>>>>>>> the categorization
>>> but
>>>>> not
>>>>>>>>>>> very
>>>>>>>>>>>> sure it is a good choice. Does any other
>>>>>>>>>>>> folks provide opinion?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi, folks,
>>>>>>>>>>>>
>>>>>>>>>>>> Just update the KIP-216, please take a look.
>>>>>>>>>>>>
>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng
>>>>>>>>>>>> <vi...@is-land.com.tw>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, folks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you suggestion, really appreciate it.
>>>>>>>>>>>>> :) I understand your concern. I'll merge
>>> StreamsNotRunningException
>>>>> and
>>>>>>>>>>>>> StateStoreNotAvailableException.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John
>>>>>>>>>>>>> Roesler <
>>> vvcephei@apache.org
>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Vito,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, thanks for the KIP. Sorry the
>>>>>>>>>>>>>> discussion has been so
>>> long.
>>>>>>>>>>>>>> Hopefully, we can close it out soon.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I agree we can drop
>>>>>>>>>>>>>> StreamsNotRunningException in favor of
>>>>>>>>>>>>>> just StateStoreNotAvailableException.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unfortunately, I have some higher-level
>>>>>>>>>>>>>> concerns. The value of these exceptions
>>>>>>>>>>>>>> is that they tell you how to handle the
>>>>>>>>>>>>>> various situations that can arise while
>>>>>>>>>>>>>> querying a distributed data store.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ideally, as a caller, I should be able to
>>>>>>>>>>>>>> just catch
>>> "retriable"
>>>>> or
>>>>>>>>>>>>>> "fatal" and handle them appropriately.
>>>>>>>>>>>>>> Otherwise, there's no point in having
>>>>>>>>>>>>>> categories, and we should just have all
>>>>>>>>>>>>>> the exceptions extend
>>>>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Presently, it's not possible to tell from
>>>>>>>>>>>>>> just the "retriable"/"fatal" distinction
>>>>>>>>>>>>>> what to do. You can tell from the
>>>>>>>>>>>>>> descriptions of the various exceptions.
>>>>>>>>>>>>>> E.g.:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Retriable: * StreamsRebalancingException:
>>>>>>>>>>>>>> the exact same call should just be
>>>>>>>>>>>>>> retried until the rebalance is complete *
>>>>>>>>>>>>>> StateStoreMigratedException: the store
>>>>>>>>>>>>>> handle is now invalid, so you need to
>>>>>>>>>>>>>> re-discover the instance and get a new
>>>>>>>>>>>>>> handle on that instance. In other words,
>>>>>>>>>>>>>> the query itself may be valid, but the
>>>>>>>>>>>>>> particular method invocation on this
>>>>>>>>>>>>>> particular instance has encountered a
>>>>>>>>>>>>>> fatal exception.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fatal: * UnknownStateStoreException: this
>>>>>>>>>>>>>> is truly fatal. No amount of retrying or
>>>>>>>>>>>>>> re-discovering is going to get you a
>>>>>>>>>>>>>> handle
>>>>> on a
>>>>>>>>>>>>>> store that doesn't exist in the cluster.
>>>>>>>>>>>>>> * StateStoreNotAvailableException: this
>>>>>>>>>>>>>> is actually
>>> recoverable,
>>>>>>>>>>>>>> since the store might exist in the
>>>>>>>>>>>>>> cluster, but isn't
>>>>> available
>>>>>>>>> on
>>>>>>>>>>>>>> this particular instance (which is shut
>>>>>>>>>>>>>> down or whatever).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Personally, I'm not a fan of code
>>>>>>>>>>>>>> bureaucracy, so I'm 100%
>>> fine
>>>>>>>>>>>>>> with omitting the categorization and just
>>>>>>>>>>>>>> having 5 subclasses of
>>>>>>>>>>>>>> InvalidStateStoreException. Each of them
>>>>>>>>>>>>>> would tell you how to handle them, and
>>>>>>>>>>>>>> it's not too many to really understand
>>>>>>>>>>>>>> and handle each one.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you really want to have a middle tier,
>>>>>>>>>>>>>> I'd recommend: *
>>>>>>>>>>>>>> RetryableStateStoreException: the exact
>>>>>>>>>>>>>> same call should be repeated. *
>>>>>>>>>>>>>> RecoverableStateStoreException: the store
>>>>>>>>>>>>>> handle should be discarded and the caller
>>>>>>>>>>>>>> should re-discover the location of the
>>>>>>>>>>>>>> store and repeat the query on the correct
>>>>>>>>>>>>>> instance. * FatalStateStoreException: the
>>>>>>>>>>>>>> query/request is totally invalid and will
>>>>>>>>>>>>>> never succeed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, attempting to categorize the
>>>>>>>>>>>>>> proposed exceptions reveals even problems
>>>>>>>>>>>>>> with this categorization: Retriable: *
>>>>>>>>>>>>>> StreamsRebalancingException Recoverable:
>>>>>>>>>>>>>> * StateStoreMigratedException *
>>>>>>>>>>>>>> StreamsNotRunningException Fatal: *
>>>>>>>>>>>>>> UnknownStateStoreException
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But StreamsNotStartedException is
>>>>>>>>>>>>>> strange... It means that one code path
>>>>>>>>>>>>>> got a handle on a specific KafkaStreams
>>>>>>>>>>>>>> object instance and sent it a query
>>>>>>>>>>>>>> before another code path invoked the
>>>>>>>>>>>>>> start() method on the exact same object
>>>>>>>>>>>>>> instance. It seems like the most likely
>>>>>>>>>>>>>> scenario is that whoever wrote the
>>>>>>>>>>>>>> program just forgot to call start()
>>>>>>>>>>>>>> before querying, in which case, retrying
>>>>>>>>>>>>>> isn't going to help, and a fatal
>>> exception
>>>>>>>>>>>>>> is more appropriate. I.e., it sounds like
>>>>>>>>>>>>>> a "first 15 minutes experience" problem,
>>>>>>>>>>>>>> and making it fatal would be more
>>>>>>>>>>>>>> helpful. Even in a production context,
>>>>>>>>>>>>>> there's no reason not to sequence your
>>>>>>>>>>>>>> application startup such that you don't
>>>>>>>>>>>>>> accept queries until after Streams is
>>>>>>>>>>>>>> started. Thus, I guess I'd categorize it
>>>>>>>>>>>>>> under "fatal".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regardless of whether you make it fatal
>>>>>>>>>>>>>> or retriable, you'd still have a whole
>>>>>>>>>>>>>> category with only one exception in it,
>>>>>>>>>>>>>> and the other two categories only have
>>>>>>>>>>>>>> two exceptions. Plus, as you pointed out
>>>>>>>>>>>>>> in the KIP, you can't get all exceptions
>>>>>>>>>>>>>> in all cases anyway: * store() can only
>>>>>>>>>>>>>> throw NotStarted, NotRunning, and
>>>>>>>>>>>>>> Unknown * actual store queries can only
>>>>>>>>>>>>>> throw Rebalancing, Migrated, and
>>>>>>>>>>>>>> NotRunning
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thus, in practice also, there are exactly
>>>>>>>>>>>>>> three categories and also exactly three
>>>>>>>>>>>>>> exception types. It doesn't seem like
>>>>>>>>>>>>>> there's a great advantage to the
>>>>>>>>>>>>>> categories here. To avoid the
>>>>>>>>>>>>>> categorization problem and also to
>>>>>>>>>>>>>> clarify what exceptions can actually be
>>>>>>>>>>>>>> thrown in different circumstances, it
>>>>>>>>>>>>>> seems like we should just: * get rid of
>>>>>>>>>>>>>> the middle tier and make all the
>>>>>>>>>>>>>> exceptions extend
>>>>>>>>>>>>>> InvalidStateStoreException * drop
>>>>>>>>>>>>>> StateStoreNotAvailableException in favor
>>>>>>>>>>>>>> of StreamsNotRunningException * clearly
>>>>>>>>>>>>>> document on all public methods which
>>>>>>>>>>>>>> exceptions need to be handled
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How do you feel about this? Thanks,
>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill
>>>>>>>>>>>>>> Bejeck wrote:
>>>>>>>>>>>>>>> Thanks for KIP Vito.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Overall the KIP LGTM, but I'd have to
>>>>>>>>>>>>>>> agree with others on
>>>>> merging
>>>>>>>>>>> the
>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Since in both cases, the thread state
>>>>>>>>>>>>>>> is in
>>> `PENDING_SHUTDOWN ||
>>>>>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure
>>>>>>>>>>>>>>> how we could
>>> distinguish
>>>>>>>>>>> when to
>>>>>>>>>>>>>>> use the different exceptions. Maybe a
>>>>>>>>>>>>>>> good middle ground would be to have a
>>>>> detailed
>>>>>>>>>>>>>>> exception message.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The KIP freeze is close, so I think if
>>>>>>>>>>>>>>> we can agree on this,
>>> we
>>>>> can
>>>>>>>>>>>>>> wrap up
>>>>>>>>>>>>>>> the voting soon.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks, Bill
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM
>>>>>>>>>>>>>>> Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's still unclear to me what the
>>>>>>>>>>>>>>>> advantage is, to have both
>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For both cased, the state is
>>>>>>>>>>>>>>>> `PENDING_SHUTDOWN /
>>> NOT_RUNNING /
>>>>>>>>>>> ERROR`
>>>>>>>>>>>>>>>> and thus, for a user point of view,
>>>>>>>>>>>>>>>> why does it matter if
>>> the
>>>>>>>>>>> store is
>>>>>>>>>>>>>>>> closed on not? I don't understand
>>>>>>>>>>>>>>>> why/how this information
>>>>> would
>>>>>>>>>>> be
>>>>>>>>>>>>>>>> useful? Do you have a concrete
>>>>>>>>>>>>>>>> example in mind how a user
>>> would
>>>>>>>>>>> react
>>>>>>>>>>>>>>>> differently to both exceptions?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> @Vinoth: about
>>>>>>>>>>>>>>>> `StreamsRebalancingException` -- to
>>>>>>>>>>>>>>>> me, it
>>> seems
>>>>>>>>>>> best
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> actually do this on a per-query
>>>>>>>>>>>>>>>> basis, ie, have an overload
>>>>>>>>>>>>>>>> `KafkaStreams#store(...)` that takes
>>>>>>>>>>>>>>>> a boolean flag that
>>> allow
>>>>> to
>>>>>>>>>>>>>>>> _disable_ the exception and opt-in to
>>>>>>>>>>>>>>>> query a active store
>>>>> during
>>>>>>>>>>>>>>>> recovery. However, as KIP-535
>>>>>>>>>>>>>>>> actually introduces this
>>> change
>>>>> in
>>>>>>>>>>>>>>>> behavior, I think KIP-216 should not
>>>>>>>>>>>>>>>> cover this, but KIP-535
>>>>>>>>>>> should be
>>>>>>>>>>>>>>>> updated. I'll follow up on the other
>>>>>>>>>>>>>>>> KIP thread to raise
>>> this
>>>>>>>>>>> point.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Hi, Matthias & Vinoth,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What is still unclear to me is,
>>>>>>>>>>>>>>>>>> what we gain by having
>>> both
>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>> `StateStoreNotAvailableException`. Both
>>>>>>>>>>>>>>>>>> exception are thrown when
>>>>>>>>>>>>>>>>>> KafkaStreams is in state
>>>>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a
>>>>>>>>>>>>>>>>>> user what do I gain to
>>> know
>>>>>>>>>>> if the
>>>>>>>>>>>>>>>>>> state store is closed on not -- I
>>>>>>>>>>>>>>>>>> can't query it anyway?
>>>>> Maybe
>>>>>>>>>>> I
>>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, both
>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>>>>> are fatal exception. But
>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`
>>>>>>>>>>>>>>>>> is fatal exception
>>> about
>>>>>>>>>>> state
>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>> related. I think it would be
>>>>>>>>>>>>>>>>> helpful that if user need to
>>> distinguish
>>>>>>>>>>> these
>>>>>>>>>>>>>> two
>>>>>>>>>>>>>>>>> different case to handle it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm not very sure, does that make
>>>>>>>>>>>>>>>>> sense?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM
>>>>>>>>>>>>>>>>> Vinoth Chandar <
>>>>>>>>>>> vinoth@apache.org>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> +1 on merging
>>>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`,
>>>>>>>>>>>>>>>>>> both exceptions are
>>> fatal
>>>>>>>>>>>>>> anyway. IMO
>>>>>>>>>>>>>>>>>> its best to have these exceptions
>>>>>>>>>>>>>>>>>> be about the state store
>>>>>>>>>>> (and not
>>>>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>>>>> state), to easier understanding.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Additionally, KIP-535 allows for
>>>>>>>>>>>>>>>>>> querying of state stores
>>> in
>>>>>>>>>>>>>> rebalancing
>>>>>>>>>>>>>>>>>> state. So do we need the
>>>>>>>>>>>>>>>>>> StreamsRebalancingException?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias
>>>>>>>>>>>>>>>>>> J. Sax" <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Sorry that I dropped the ball
>>>>>>>>>>>>>>>>>>> on this...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP.
>>>>>>>>>>>>>>>>>>> Overall LGTM now. Feel free
>>> to
>>>>>>>>>>> start
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> VOTE
>>>>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What is still unclear to me is,
>>>>>>>>>>>>>>>>>>> what we gain by having
>>> both
>>>>>>>>>>>>>>>>>>> `StreamsNotRunningException`
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> `StateStoreNotAvailableException`.
>>>>>>>>>>>>>>>> Both
>>>>>>>>>>>>>>>>>>> exception are thrown when
>>>>>>>>>>>>>>>>>>> KafkaStreams is in state
>>>>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as
>>>>>>>>>>>>>>>>>>> a user what do I gain to
>>> know
>>>>>>>>>>> if
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> state store is closed on not --
>>>>>>>>>>>>>>>>>>> I can't query it anyway?
>>>>>>>>>>> Maybe I
>>>>>>>>>>>>>> miss
>>>>>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Sorry for the late reply,
>>>>>>>>>>>>>>>>>>>> thanks for the review.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
Why is it only thrown if the state is REBALANCING? A
>>> store
>>>>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>>> migrated during a
>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
>>>>>>>>>>>>>>>>>>>>> Streams might
>>>>> resume
>>>>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>>>> RUNNING state and afterward
>>>>>>>>>>>>>>>>>>>>> somebody tries to use an
>>> old
>>>>>>>>>>> store
>>>>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>>>>> Also, if state is
>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
>>>>>>>>>>>>>>>>>>>>> Hence, I think
>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>>>>> does only make sense
>>> during
>>>>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you point this, already
>>>>>>>>>>>>>>>>>>>> updated.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Why do we need to distinguish
>>>>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
`KafkaStreamsNotRunningException` may be caused by
>>> various
>>>>>>>>>>>>>> reasons, I
>>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>>>>> it would be helpful that the
>>>>>>>>>>>>>>>>>>>> user can distinguish whether
>>>>>>>>>>>>>>>>>>>> it is caused by the state
>>>>> store
>>>>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>>>>> (Maybe I am wrong...)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Last, why do we distinguish
>>>>>>>>>>>>>>>>>>>> between `KafkaStreams`
>>> instance
>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it
>>>>>>>>>>>>>>>>>>>>> seems we should always
>>> refer to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>>>>> because that is the level
>>>>>>>>>>>>>>>>>>>>> of granularity in which we
>>>>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>>>>> IQ atm.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Totally agree. Do you mean
>>>>>>>>>>>>>>>>>>>> the naming of state store
>>>>>>>>>>> exceptions?
>>>>>>>>>>>>>>>>>>>> I don't have special reason
>>>>>>>>>>>>>>>>>>>> to distinguish these two.
>>>>>>>>>>>>>>>>>>>> Your suggestion look more
>>>>>>>>>>>>>>>>>>>> reasonable for the exception
>>>>>>>>>>> naming.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Last, for
>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
>>>>>>>>>>>>>>>>>>>> I would add
>>> that a
>>>>>>>>>>> user
>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> rediscover the store and
>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
>>>>>>>>>>>>>>>>>>>>> the
>>> store
>>>>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>>>>> invalid and a new store
>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
>>>>>>>>>>>>>>>>>>>>> That
>>> is
>>>>> a
>>>>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
>>>>>>>>>>>>>>>>>>>>> that allows for
>>>>>>>>>>> "blind"
>>>>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>>>>> that either resolve (if the
>>>>>>>>>>>>>>>>>>>>> store is still on the same
>>>>>>>>>>> instance
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
>>>>>>>>>>>>>>>>>>>>> changes to
>>>>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> store was migrated away
>>>>>>>>>>>>>>>>>>>>> during rebalancing).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The KIP already updated,
>>>>>>>>>>>>>>>>>>>> please take a look. :)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48
>>>>>>>>>>>>>>>>>>>> PM Matthias J. Sax <
>>>>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Any update on this KIP?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM,
>>>>>>>>>>>>>>>>>>>>> Matthias J. Sax wrote:
>>>>>>>>>>>>>>>>>>>>>> Sorry for the late reply.
>>>>>>>>>>>>>>>>>>>>>> The 2.4 deadline kept us
>>> quite
>>>>>>>>>>> busy.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
Why is it only thrown if the state is REBALANCING? A
>>>>> store
>>>>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>>>>> migrated during a
>>>>>>>>>>>>>>>>>>>>>> rebalance, and Kafka
>>>>>>>>>>>>>>>>>>>>>> Streams might
>>>>> resume
>>>>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>>>>> RUNNING state and
>>>>>>>>>>>>>>>>>>>>>> afterward somebody tries
>>>>>>>>>>>>>>>>>>>>>> to use an
>>> old
>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>>>>>> Also, if state is
>>>>>>>>>>>>>>>>>>>>>> REBALANCING, should we
>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`?
>>>>>>>>>>>>>>>>>>>>>> Hence, I think
>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>>>>>> does only make sense
>>> during
>>>>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Why do we need to
>>>>>>>>>>>>>>>>>>>>>> distinguish between
>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Last, why do we
>>>>>>>>>>>>>>>>>>>>>> distinguish between
>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams`
>>>>>>>>>>> instance and
>>>>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me,
>>>>>>>>>>>>>>>>>>>>>> it seems we should
>>>>>>>>>>>>>>>>>>>>>> always
>>> refer
>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>>>>>> because that is the level
>>>>>>>>>>>>>>>>>>>>>> of granularity in which
>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>>>>> IQ
>>>>>>>>>>>>>>>>>>>>> atm.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Last, for
>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`,
>>>>>>>>>>>>>>>>>>>>>> I would add
>>>>> that a
>>>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>>>>> rediscover the store and
>>>>>>>>>>>>>>>>>>>>>> cannot blindly retry as
>>>>>>>>>>>>>>>>>>>>>> the
>>>>> store
>>>>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>>>>>> invalid and a new store
>>>>>>>>>>>>>>>>>>>>>> handle must be retrieved.
>>>>>>>>>>>>>>>>>>>>>> That
>>>>> is a
>>>>>>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`
>>>>>>>>>>>>>>>>>>>>>> that allows for
>>>>>>>>>>> "blind"
>>>>>>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>>>>>> that either resolve (if
>>>>>>>>>>>>>>>>>>>>>> the store is still on the
>>>>>>>>>>>>>>>>>>>>>> same
>>>>>>>>>>> instance
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>> rebalancing finishes, or
>>>>>>>>>>>>>>>>>>>>>> changes to
>>>>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> store was migrated away
>>>>>>>>>>>>>>>>>>>>>> during rebalancing).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito
>>>>>>>>>>>>>>>>>>>>>> Jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>> My bad. The short link
>>>>>>>>>>>>>>>>>>>>>>> `https://shorturl.at/CDNT9`
>>>
>>>>>>>>>>>>>>>>>>>>>>>
<https://shorturl.at/CDNT9>
>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
<https://shorturl.at/CDNT9> seems incorrect.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Please use the
>>>>>>>>>>>>>>>>>>>>>>> following instead:
>>>>>>>>>>> https://shorturl.at/bkKQU
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at
>>>>>>>>>>>>>>>>>>>>>>> 10:53 AM Vito Jeng <
>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
Thank you for explanation. I agree with your
>>> opinion.
>>>>>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()`
>>>>>>>>>>>>>>>>>>>>>>>> would never throw
>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
For the case that corresponding thread crashes
>>> after we
>>>>>>>>>>>>>> handed out
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> store handle. We may
>>>>>>>>>>>>>>>>>>>>>>>> throw
>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
In `StreamThreadStateStoreProvider`, we would throw
>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>> when stream
>>> thread is
>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>> running(
>>>>>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9)
>>>>>>>>>>>>>>>>>>>>>>>> or throw
>>>>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>>>>> store is
>>>>>>>>>>>>>>>>>>>>>>>> closed(https://shorturl.at/hrvAN).
>>>>>>>>>>>>>>>>>>>>>>>> So I
>>> think
>>>>>>>>>>> we
>>>>>>>>>>>>>> do not
>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> add a new type for
>>>>>>>>>>>>>>>>>>>>>>>> this case. Does that
>>>>>>>>>>>>>>>>>>>>>>>> make sense?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> About
>>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>>>>>>>> vs
>>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
I understand your point. I rename
>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
About check unknown state store names:
>>>>>>>>>>>>>>>>>>>>>>>> Thank you for the
>>>>>>>>>>>>>>>>>>>>>>>> hint. I add a new
>>>>>>>>>>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>>>>> `UnknownStateStoreException`
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>> this case.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Also, we should
>>>>>>>>>>>>>>>>>>>>>>>>> still have fatal
>>>>>>>>>>>>>>>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>>>>>> Not sure why you
>>>>>>>>>>> remove
>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thank you point this,
>>>>>>>>>>>>>>>>>>>>>>>> already add it
>>>>>>>>>>>>>>>>>>>>>>>> again.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The KIP already
>>>>>>>>>>>>>>>>>>>>>>>> updated, please take
>>>>>>>>>>>>>>>>>>>>>>>> a look.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> --- Vito
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl45vCwACgkQO4miYXKq
/OhT4Q//elvNj2BRIMd7SqyQhpa6sSq4k2iG3wFlr/B2xkmkZPWUICt3SpCisIxG
RlP5ml7Mi8IcWv9jmMux9C1NpftboLudxIUcun/I4cP0T3S7ytXfzdO+V1WuD9xe
hfeqMwJTytusEv5VO3MyY+jYGqmPJrPJRViZ2Uwsj3Ojk0SZRb/m+b6ah6BGMh0E
nlUIux7cabg1OZ/ee0x21hcAB32lVIRfPLKZeHCTHuYmlW76X4pXhFxOm69Pumtj
ciVMr/pl7B+XiMk0C2Po04zQcP8+5/O7LYU4e8ha12NtuNuDTlQLGUb+S9Qrbxb2
xdFrlC81RROmIdIOTLiJeeBqhzd19llDgMhBf4spJMOeLn7359PX6r/9tDqonJrQ
wpSp4S54MDYfTPHUPqA6u/FsE0BX4EMU71ckC9rXVRRUnQ8A8cMk/6qWlRu8NvxQ
IiYuSo48UENlcTHxXC/rzIDp54gvtv2iml8QcOg6tS+hzhAI4yoY+1w4pnjPmRo0
4D1hnq5bd1SLrGSGcCxVbN0jtwgkcr50HBb1UkIDRndqapfKwZOMV65tIwTvxaaZ
r2QPYyPd6ZDgeMdy0r94z0SMwatqGCmJD9EQmSmLBulemrvYxPGUevQls+WgE4WE
zc3LtoyfdlXlkttTU0Q/dm2H9OU4s3QO+6PEdC+Qwi3aCKKhdUQ=
=4EMC
-----END PGP SIGNATURE-----
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, folks,
KIP-562(KAFKA-9445) already merged three days ago.
I have updated KIP-216 to reflect the KIP-562.
The main change is to introduce a new exception
`InvalidStateStorePartitionException`, will be thrown when user requested
partition not available.
Please take a look and any feedback is welcome.
Thanks Matthias for the reminder.
---
Vito
On Thu, Jan 23, 2020 at 2:13 PM Vito Jeng <vi...@is-land.com.tw> wrote:
> Got it, thanks Matthias.
>
> ---
> Vito
>
>
> On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Thanks Vito.
>>
>> I am also ok with either name. Just a personal slight preference, but
>> not a important.
>>
>>
>> -Matthias
>>
>> On 1/21/20 6:52 PM, Vito Jeng wrote:
>> > Thanks Matthias.
>> >
>> > The KIP is about InvalidStateStoreException.
>> > I pick `StateStoreNotAvailableException` because it may be more
>> intuitive
>> > than `StreamsNotRunningException`.
>> >
>> > No matter which one picked, it's good to me.
>> >
>> > ---
>> > Vito
>> >
>> >
>> > On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax <ma...@confluent.io>
>> > wrote:
>> >
>> >> Thanks for updating the KIP!
>> >>
>> >> One last comment/question: you kept `StateStoreNotAvailableException`
>> in
>> >> favor of `StreamsNotRunningException` (to merge both as suggested).
>> >>
>> >> I am wondering, if it might be better to keep
>> >> `StreamsNotRunningException` instead of
>> >> `StateStoreNotAvailableException`, because this exception is thrown if
>> >> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
>> >>
>> >>
>> >>
>> >> -Matthias
>> >>
>> >> On 1/17/20 9:56 PM, John Roesler wrote:
>> >>> Thanks, Vito. I've just cast my vote.
>> >>> -John
>> >>>
>> >>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>> >>>> Hi, folks,
>> >>>>
>> >>>> Just update the KIP, please take a look.
>> >>>>
>> >>>> Thanks!
>> >>>>
>> >>>> ---
>> >>>> Vito
>> >>>>
>> >>>>
>> >>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng <vi...@is-land.com.tw>
>> wrote:
>> >>>>
>> >>>>> Thanks Bill, John and Matthias. Glad you guys joined this
>> discussion.
>> >>>>> I got a lot out of the discussion.
>> >>>>>
>> >>>>> I would like to update KIP-216 base on John's suggestion to remove
>> the
>> >>>>> category.
>> >>>>>
>> >>>>>
>> >>>>> ---
>> >>>>> Vito
>> >>>>>
>> >>>>>
>> >>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
>> matthias@confluent.io
>> >>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>>> Nevertheless, if we omit the categorization, it’s moot.
>> >>>>>>
>> >>>>>> Ack.
>> >>>>>>
>> >>>>>> I am fine to remove the middle tier. As John pointed out, it might
>> be
>> >>>>>> weird to have only one concrete exception type per category. We can
>> >> also
>> >>>>>> explain in detail how to handle each exception in their JavaDocs.
>> >>>>>>
>> >>>>>>
>> >>>>>> -Matthias
>> >>>>>>
>> >>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>> >>>>>>> Vito,
>> >>>>>>>
>> >>>>>>> Thanks for the updates, the KIP LGTM.
>> >>>>>>>
>> >>>>>>> -Bill
>> >>>>>>>
>> >>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <
>> vvcephei@apache.org>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Vito,
>> >>>>>>>>
>> >>>>>>>> Haha, your archive game is on point!
>> >>>>>>>>
>> >>>>>>>> What Matthias said in that email is essentially what I figured
>> was
>> >> the
>> >>>>>>>> rationale. It makes sense, but the point I was making is that
>> this
>> >>>>>> really
>> >>>>>>>> doesn’t seem like a good way to structure a production app. On
>> the
>> >>>>>> other
>> >>>>>>>> hand, considering the exception fatal has a good chance of
>> avoiding
>> >> a
>> >>>>>>>> frustrating debug session if you just forgot to call start.
>> >>>>>>>>
>> >>>>>>>> Nevertheless, if we omit the categorization, it’s moot.
>> >>>>>>>>
>> >>>>>>>> It would be easy to add a categorization layer later if we want
>> it,
>> >> but
>> >>>>>>>> not very easy to change it if we get it wrong.
>> >>>>>>>>
>> >>>>>>>> Thanks for your consideration!
>> >>>>>>>> -John
>> >>>>>>>>
>> >>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
>> >>>>>>>>> Hi John,
>> >>>>>>>>>
>> >>>>>>>>> About `StreamsNotStartedException is strange` --
>> >>>>>>>>> The original idea came from Matthias, two years ago. :)
>> >>>>>>>>> You can reference here:
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>
>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>> >>>>>>>>>
>> >>>>>>>>> About omitting the categorization --
>> >>>>>>>>> It looks reasonable. I'm fine with omitting the categorization
>> but
>> >> not
>> >>>>>>>> very
>> >>>>>>>>> sure it is a good choice.
>> >>>>>>>>> Does any other folks provide opinion?
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Hi, folks,
>> >>>>>>>>>
>> >>>>>>>>> Just update the KIP-216, please take a look.
>> >>>>>>>>>
>> >>>>>>>>> ---
>> >>>>>>>>> Vito
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw>
>> >>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Hi, folks,
>> >>>>>>>>>>
>> >>>>>>>>>> Thank you suggestion, really appreciate it. :)
>> >>>>>>>>>> I understand your concern. I'll merge
>> StreamsNotRunningException
>> >> and
>> >>>>>>>>>> StateStoreNotAvailableException.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> ---
>> >>>>>>>>>> Vito
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <
>> vvcephei@apache.org
>> >>>
>> >>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Hey Vito,
>> >>>>>>>>>>>
>> >>>>>>>>>>> Yes, thanks for the KIP. Sorry the discussion has been so
>> long.
>> >>>>>>>>>>> Hopefully, we can close it out soon.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I agree we can drop StreamsNotRunningException in favor of
>> >>>>>>>>>>> just StateStoreNotAvailableException.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Unfortunately, I have some higher-level concerns. The value
>> >>>>>>>>>>> of these exceptions is that they tell you how to handle the
>> >>>>>>>>>>> various situations that can arise while querying a distributed
>> >>>>>>>>>>> data store.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Ideally, as a caller, I should be able to just catch
>> "retriable"
>> >> or
>> >>>>>>>>>>> "fatal" and handle them appropriately. Otherwise, there's no
>> >>>>>>>>>>> point in having categories, and we should just have all the
>> >>>>>>>>>>> exceptions extend InvalidStateStoreException.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Presently, it's not possible to tell from just the
>> >>>>>>>>>>> "retriable"/"fatal" distinction what to do. You can tell
>> >>>>>>>>>>> from the descriptions of the various exceptions. E.g.:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Retriable:
>> >>>>>>>>>>> * StreamsRebalancingException: the exact same call
>> >>>>>>>>>>> should just be retried until the rebalance is complete
>> >>>>>>>>>>> * StateStoreMigratedException: the store handle is
>> >>>>>>>>>>> now invalid, so you need to re-discover the instance
>> >>>>>>>>>>> and get a new handle on that instance. In other words,
>> >>>>>>>>>>> the query itself may be valid, but the particular method
>> >>>>>>>>>>> invocation on this particular instance has encountered
>> >>>>>>>>>>> a fatal exception.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Fatal:
>> >>>>>>>>>>> * UnknownStateStoreException: this is truly fatal. No amount
>> >>>>>>>>>>> of retrying or re-discovering is going to get you a handle
>> >> on a
>> >>>>>>>>>>> store that doesn't exist in the cluster.
>> >>>>>>>>>>> * StateStoreNotAvailableException: this is actually
>> recoverable,
>> >>>>>>>>>>> since the store might exist in the cluster, but isn't
>> >> available
>> >>>>>> on
>> >>>>>>>>>>> this particular instance (which is shut down or whatever).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100%
>> fine
>> >>>>>>>>>>> with omitting the categorization and just having 5 subclasses
>> >>>>>>>>>>> of InvalidStateStoreException. Each of them would tell you
>> >>>>>>>>>>> how to handle them, and it's not too many to really
>> >>>>>>>>>>> understand and handle each one.
>> >>>>>>>>>>>
>> >>>>>>>>>>> If you really want to have a middle tier, I'd recommend:
>> >>>>>>>>>>> * RetryableStateStoreException: the exact same call
>> >>>>>>>>>>> should be repeated.
>> >>>>>>>>>>> * RecoverableStateStoreException: the store handle
>> >>>>>>>>>>> should be discarded and the caller should re-discover
>> >>>>>>>>>>> the location of the store and repeat the query on the
>> >>>>>>>>>>> correct instance.
>> >>>>>>>>>>> * FatalStateStoreException: the query/request is totally
>> >>>>>>>>>>> invalid and will never succeed.
>> >>>>>>>>>>>
>> >>>>>>>>>>> However, attempting to categorize the proposed exceptions
>> >>>>>>>>>>> reveals even problems with this categorization:
>> >>>>>>>>>>> Retriable:
>> >>>>>>>>>>> * StreamsRebalancingException
>> >>>>>>>>>>> Recoverable:
>> >>>>>>>>>>> * StateStoreMigratedException
>> >>>>>>>>>>> * StreamsNotRunningException
>> >>>>>>>>>>> Fatal:
>> >>>>>>>>>>> * UnknownStateStoreException
>> >>>>>>>>>>>
>> >>>>>>>>>>> But StreamsNotStartedException is strange... It means that
>> >>>>>>>>>>> one code path got a handle on a specific KafkaStreams object
>> >>>>>>>>>>> instance and sent it a query before another code path
>> >>>>>>>>>>> invoked the start() method on the exact same object instance.
>> >>>>>>>>>>> It seems like the most likely scenario is that whoever wrote
>> >>>>>>>>>>> the program just forgot to call start() before querying, in
>> >>>>>>>>>>> which case, retrying isn't going to help, and a fatal
>> exception
>> >>>>>>>>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
>> >>>>>>>>>>> experience" problem, and making it fatal would be more
>> >>>>>>>>>>> helpful. Even in a production context, there's no reason not
>> >>>>>>>>>>> to sequence your application startup such that you don't
>> >>>>>>>>>>> accept queries until after Streams is started. Thus, I guess
>> >>>>>>>>>>> I'd categorize it under "fatal".
>> >>>>>>>>>>>
>> >>>>>>>>>>> Regardless of whether you make it fatal or retriable, you'd
>> >>>>>>>>>>> still have a whole category with only one exception in it,
>> >>>>>>>>>>> and the other two categories only have two exceptions.
>> >>>>>>>>>>> Plus, as you pointed out in the KIP, you can't get all
>> >>>>>>>>>>> exceptions in all cases anyway:
>> >>>>>>>>>>> * store() can only throw NotStarted, NotRunning,
>> >>>>>>>>>>> and Unknown
>> >>>>>>>>>>> * actual store queries can only throw Rebalancing,
>> >>>>>>>>>>> Migrated, and NotRunning
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thus, in practice also, there are exactly three categories
>> >>>>>>>>>>> and also exactly three exception types. It doesn't seem
>> >>>>>>>>>>> like there's a great advantage to the categories here. To
>> >>>>>>>>>>> avoid the categorization problem and also to clarify what
>> >>>>>>>>>>> exceptions can actually be thrown in different circumstances,
>> >>>>>>>>>>> it seems like we should just:
>> >>>>>>>>>>> * get rid of the middle tier and make all the exceptions
>> >>>>>>>>>>> extend InvalidStateStoreException
>> >>>>>>>>>>> * drop StateStoreNotAvailableException in favor of
>> >>>>>>>>>>> StreamsNotRunningException
>> >>>>>>>>>>> * clearly document on all public methods which exceptions
>> >>>>>>>>>>> need to be handled
>> >>>>>>>>>>>
>> >>>>>>>>>>> How do you feel about this?
>> >>>>>>>>>>> Thanks,
>> >>>>>>>>>>> -John
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
>> >>>>>>>>>>>> Thanks for KIP Vito.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Overall the KIP LGTM, but I'd have to agree with others on
>> >> merging
>> >>>>>>>> the
>> >>>>>>>>>>>> `StreamsNotRunningException` and
>> >> `StateStoreNotAvailableException`
>> >>>>>>>>>>> classes.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Since in both cases, the thread state is in
>> `PENDING_SHUTDOWN ||
>> >>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could
>> distinguish
>> >>>>>>>> when to
>> >>>>>>>>>>>> use the different
>> >>>>>>>>>>>> exceptions. Maybe a good middle ground would be to have a
>> >> detailed
>> >>>>>>>>>>>> exception message.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> The KIP freeze is close, so I think if we can agree on this,
>> we
>> >> can
>> >>>>>>>>>>> wrap up
>> >>>>>>>>>>>> the voting soon.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>> Bill
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
>> >>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Vito,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> It's still unclear to me what the advantage is, to have both
>> >>>>>>>>>>>>> `StreamsNotRunningException` and
>> >>>>>>>> `StateStoreNotAvailableException`?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> For both cased, the state is `PENDING_SHUTDOWN /
>> NOT_RUNNING /
>> >>>>>>>> ERROR`
>> >>>>>>>>>>>>> and thus, for a user point of view, why does it matter if
>> the
>> >>>>>>>> store is
>> >>>>>>>>>>>>> closed on not? I don't understand why/how this information
>> >> would
>> >>>>>>>> be
>> >>>>>>>>>>>>> useful? Do you have a concrete example in mind how a user
>> would
>> >>>>>>>> react
>> >>>>>>>>>>>>> differently to both exceptions?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it
>> seems
>> >>>>>>>> best
>> >>>>>>>>>>> to
>> >>>>>>>>>>>>> actually do this on a per-query basis, ie, have an overload
>> >>>>>>>>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that
>> allow
>> >> to
>> >>>>>>>>>>>>> _disable_ the exception and opt-in to query a active store
>> >> during
>> >>>>>>>>>>>>> recovery. However, as KIP-535 actually introduces this
>> change
>> >> in
>> >>>>>>>>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
>> >>>>>>>> should be
>> >>>>>>>>>>>>> updated. I'll follow up on the other KIP thread to raise
>> this
>> >>>>>>>> point.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
>> >>>>>>>>>>>>>> Hi, Matthias & Vinoth,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks for the feedback.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> What is still unclear to me is, what we gain by having
>> both
>> >>>>>>>>>>>>>>> `StreamsNotRunningException` and
>> >>>>>>>>>>> `StateStoreNotAvailableException`. Both
>> >>>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
>> >>>>>>>>>>> PENDING_SHUTDOWN /
>> >>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to
>> know
>> >>>>>>>> if the
>> >>>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
>> >> Maybe
>> >>>>>>>> I
>> >>>>>>>>>>> miss
>> >>>>>>>>>>>>>>> something thought?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Yes, both `StreamsNotRunningException` and
>> >>>>>>>>>>>>>> `StateStoreNotAvailableException` are fatal exception.
>> >>>>>>>>>>>>>> But `StateStoreNotAvailableException` is fatal exception
>> about
>> >>>>>>>> state
>> >>>>>>>>>>>>> store
>> >>>>>>>>>>>>>> related.
>> >>>>>>>>>>>>>> I think it would be helpful that if user need to
>> distinguish
>> >>>>>>>> these
>> >>>>>>>>>>> two
>> >>>>>>>>>>>>>> different case to handle it.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I'm not very sure, does that make sense?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
>> >>>>>>>> vinoth@apache.org>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> +1 on merging `StreamsNotRunningException` and
>> >>>>>>>>>>>>>>> `StateStoreNotAvailableException`, both exceptions are
>> fatal
>> >>>>>>>>>>> anyway. IMO
>> >>>>>>>>>>>>>>> its best to have these exceptions be about the state store
>> >>>>>>>> (and not
>> >>>>>>>>>>>>> streams
>> >>>>>>>>>>>>>>> state), to easier understanding.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Additionally, KIP-535 allows for querying of state stores
>> in
>> >>>>>>>>>>> rebalancing
>> >>>>>>>>>>>>>>> state. So do we need the StreamsRebalancingException?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
>> >>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>> Sorry that I dropped the ball on this...
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free
>> to
>> >>>>>>>> start
>> >>>>>>>>>>> a
>> >>>>>>>>>>>>> VOTE
>> >>>>>>>>>>>>>>>> thread.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> What is still unclear to me is, what we gain by having
>> both
>> >>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>> >>>>>>>>>>> `StateStoreNotAvailableException`.
>> >>>>>>>>>>>>> Both
>> >>>>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
>> >>>>>>>>>>> PENDING_SHUTDOWN /
>> >>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to
>> know
>> >>>>>>>> if
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
>> >>>>>>>> Maybe I
>> >>>>>>>>>>> miss
>> >>>>>>>>>>>>>>>> something thought?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>> >>>>>>>>>>>>>>>>> Sorry for the late reply, thanks for the review.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A
>> store
>> >>>>>>>>>>> might be
>> >>>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
>> >> resume
>> >>>>>>>>>>> back to
>> >>>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an
>> old
>> >>>>>>>> store
>> >>>>>>>>>>>>> handle.
>> >>>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>> >>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>> >>>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense
>> during
>> >>>>>>>>>>> `RUNNING`
>> >>>>>>>>>>>>>>> state.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Thank you point this, already updated.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Why do we need to distinguish between
>> >>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>> >>>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by
>> various
>> >>>>>>>>>>> reasons, I
>> >>>>>>>>>>>>>>> think
>> >>>>>>>>>>>>>>>>> it would be helpful that the
>> >>>>>>>>>>>>>>>>> user can distinguish whether it is caused by the state
>> >> store
>> >>>>>>>>>>> closed.
>> >>>>>>>>>>>>>>>>> (Maybe I am wrong...)
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
>> instance
>> >>>>>>>> and
>> >>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always
>> refer to
>> >>>>>>>> the
>> >>>>>>>>>>>>>>> instance,
>> >>>>>>>>>>>>>>>>>> because that is the level of granularity in which we
>> >>>>>>>>>>> enable/disable
>> >>>>>>>>>>>>>>> IQ atm.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Totally agree. Do you mean the naming of state store
>> >>>>>>>> exceptions?
>> >>>>>>>>>>>>>>>>> I don't have special reason to distinguish these two.
>> >>>>>>>>>>>>>>>>> Your suggestion look more reasonable for the exception
>> >>>>>>>> naming.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add
>> that a
>> >>>>>>>> user
>> >>>>>>>>>>> need
>> >>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the
>> store
>> >>>>>>>>>>> handle is
>> >>>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That
>> is
>> >> a
>> >>>>>>>>>>>>> difference
>> >>>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>> >>>>>>>> "blind"
>> >>>>>>>>>>> retries
>> >>>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
>> >>>>>>>> instance
>> >>>>>>>>>>> after
>> >>>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
>> >>>>>>>>>>> `StateStoreMigratedException` if
>> >>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> The KIP already updated, please take a look. :)
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
>> >>>>>>>>>>>>> matthias@confluent.io
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Any update on this KIP?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>> >>>>>>>>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us
>> quite
>> >>>>>>>> busy.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A
>> >> store
>> >>>>>>>>>>> might be
>> >>>>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
>> >> resume
>> >>>>>>>>>>> back to
>> >>>>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an
>> old
>> >>>>>>>> store
>> >>>>>>>>>>>>>>> handle.
>> >>>>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>> >>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>> >>>>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense
>> during
>> >>>>>>>>>>> `RUNNING`
>> >>>>>>>>>>>>>>>>>> state.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Why do we need to distinguish between
>> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>> >>>>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
>> >>>>>>>> instance and
>> >>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always
>> refer
>> >> to
>> >>>>>>>> the
>> >>>>>>>>>>>>>>> instance,
>> >>>>>>>>>>>>>>>>>>> because that is the level of granularity in which we
>> >>>>>>>>>>> enable/disable
>> >>>>>>>>>>>>>>> IQ
>> >>>>>>>>>>>>>>>>>> atm.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add
>> >> that a
>> >>>>>>>>>>> user
>> >>>>>>>>>>>>>>> need to
>> >>>>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the
>> >> store
>> >>>>>>>>>>> handle is
>> >>>>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That
>> >> is a
>> >>>>>>>>>>>>>>> difference
>> >>>>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>> >>>>>>>> "blind"
>> >>>>>>>>>>>>> retries
>> >>>>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
>> >>>>>>>> instance
>> >>>>>>>>>>>>> after
>> >>>>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
>> >>>>>>>>>>> `StateStoreMigratedException` if
>> >>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>> >>>>>>>>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9>
>> >> <https://shorturl.at/CDNT9>
>> >>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Please use the following instead:
>> >>>>>>>> https://shorturl.at/bkKQU
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
>> >>>>>>>>>>> vito@is-land.com.tw>
>> >>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Thank you for explanation. I agree with your
>> opinion.
>> >>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>> >>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> For the case that corresponding thread crashes
>> after we
>> >>>>>>>>>>> handed out
>> >>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>> store handle. We may throw
>> >>>>>>>> `KafkaStreamsNotRunningException`
>> >>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
>> >>>>>>>>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>> >>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream
>> thread is
>> >>>>>>>> not
>> >>>>>>>>>>>>>>> running(
>> >>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
>> >>>>>>>>>>> `StateStoreMigratedException`
>> >>>>>>>>>>>>>>> when
>> >>>>>>>>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I
>> think
>> >>>>>>>> we
>> >>>>>>>>>>> do not
>> >>>>>>>>>>>>>>> need
>> >>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
>> >>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> I understand your point. I rename
>> >>>>>>>>>>>>>>> `StreamThreadNotRunningException` to
>> >>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> About check unknown state store names:
>> >>>>>>>>>>>>>>>>>>>>> Thank you for the hint. I add a new type
>> >>>>>>>>>>>>>>> `UnknownStateStoreException`
>> >>>>>>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>>>>>> this case.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Also, we should still have fatal exception
>> >>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
>> >>>>>>>> remove
>> >>>>>>>>>>> it?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Thank you point this, already add it again.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> The KIP already updated, please take a look.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>
>> >>
>> >
>>
>>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Got it, thanks Matthias.
---
Vito
On Thu, Jan 23, 2020 at 1:31 PM Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks Vito.
>
> I am also ok with either name. Just a personal slight preference, but
> not a important.
>
>
> -Matthias
>
> On 1/21/20 6:52 PM, Vito Jeng wrote:
> > Thanks Matthias.
> >
> > The KIP is about InvalidStateStoreException.
> > I pick `StateStoreNotAvailableException` because it may be more intuitive
> > than `StreamsNotRunningException`.
> >
> > No matter which one picked, it's good to me.
> >
> > ---
> > Vito
> >
> >
> > On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Thanks for updating the KIP!
> >>
> >> One last comment/question: you kept `StateStoreNotAvailableException` in
> >> favor of `StreamsNotRunningException` (to merge both as suggested).
> >>
> >> I am wondering, if it might be better to keep
> >> `StreamsNotRunningException` instead of
> >> `StateStoreNotAvailableException`, because this exception is thrown if
> >> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
> >>
> >>
> >>
> >> -Matthias
> >>
> >> On 1/17/20 9:56 PM, John Roesler wrote:
> >>> Thanks, Vito. I've just cast my vote.
> >>> -John
> >>>
> >>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >>>> Hi, folks,
> >>>>
> >>>> Just update the KIP, please take a look.
> >>>>
> >>>> Thanks!
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>>
> >>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng <vi...@is-land.com.tw>
> wrote:
> >>>>
> >>>>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> >>>>> I got a lot out of the discussion.
> >>>>>
> >>>>> I would like to update KIP-216 base on John's suggestion to remove
> the
> >>>>> category.
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>>
> >>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <
> matthias@confluent.io
> >>>
> >>>>> wrote:
> >>>>>
> >>>>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>>>
> >>>>>> Ack.
> >>>>>>
> >>>>>> I am fine to remove the middle tier. As John pointed out, it might
> be
> >>>>>> weird to have only one concrete exception type per category. We can
> >> also
> >>>>>> explain in detail how to handle each exception in their JavaDocs.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >>>>>>> Vito,
> >>>>>>>
> >>>>>>> Thanks for the updates, the KIP LGTM.
> >>>>>>>
> >>>>>>> -Bill
> >>>>>>>
> >>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vvcephei@apache.org
> >
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Vito,
> >>>>>>>>
> >>>>>>>> Haha, your archive game is on point!
> >>>>>>>>
> >>>>>>>> What Matthias said in that email is essentially what I figured was
> >> the
> >>>>>>>> rationale. It makes sense, but the point I was making is that this
> >>>>>> really
> >>>>>>>> doesn’t seem like a good way to structure a production app. On the
> >>>>>> other
> >>>>>>>> hand, considering the exception fatal has a good chance of
> avoiding
> >> a
> >>>>>>>> frustrating debug session if you just forgot to call start.
> >>>>>>>>
> >>>>>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>>>>>
> >>>>>>>> It would be easy to add a categorization layer later if we want
> it,
> >> but
> >>>>>>>> not very easy to change it if we get it wrong.
> >>>>>>>>
> >>>>>>>> Thanks for your consideration!
> >>>>>>>> -John
> >>>>>>>>
> >>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> >>>>>>>>> Hi John,
> >>>>>>>>>
> >>>>>>>>> About `StreamsNotStartedException is strange` --
> >>>>>>>>> The original idea came from Matthias, two years ago. :)
> >>>>>>>>> You can reference here:
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> >>>>>>>>>
> >>>>>>>>> About omitting the categorization --
> >>>>>>>>> It looks reasonable. I'm fine with omitting the categorization
> but
> >> not
> >>>>>>>> very
> >>>>>>>>> sure it is a good choice.
> >>>>>>>>> Does any other folks provide opinion?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi, folks,
> >>>>>>>>>
> >>>>>>>>> Just update the KIP-216, please take a look.
> >>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>> Vito
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hi, folks,
> >>>>>>>>>>
> >>>>>>>>>> Thank you suggestion, really appreciate it. :)
> >>>>>>>>>> I understand your concern. I'll merge StreamsNotRunningException
> >> and
> >>>>>>>>>> StateStoreNotAvailableException.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> Vito
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <
> vvcephei@apache.org
> >>>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hey Vito,
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
> >>>>>>>>>>> Hopefully, we can close it out soon.
> >>>>>>>>>>>
> >>>>>>>>>>> I agree we can drop StreamsNotRunningException in favor of
> >>>>>>>>>>> just StateStoreNotAvailableException.
> >>>>>>>>>>>
> >>>>>>>>>>> Unfortunately, I have some higher-level concerns. The value
> >>>>>>>>>>> of these exceptions is that they tell you how to handle the
> >>>>>>>>>>> various situations that can arise while querying a distributed
> >>>>>>>>>>> data store.
> >>>>>>>>>>>
> >>>>>>>>>>> Ideally, as a caller, I should be able to just catch
> "retriable"
> >> or
> >>>>>>>>>>> "fatal" and handle them appropriately. Otherwise, there's no
> >>>>>>>>>>> point in having categories, and we should just have all the
> >>>>>>>>>>> exceptions extend InvalidStateStoreException.
> >>>>>>>>>>>
> >>>>>>>>>>> Presently, it's not possible to tell from just the
> >>>>>>>>>>> "retriable"/"fatal" distinction what to do. You can tell
> >>>>>>>>>>> from the descriptions of the various exceptions. E.g.:
> >>>>>>>>>>>
> >>>>>>>>>>> Retriable:
> >>>>>>>>>>> * StreamsRebalancingException: the exact same call
> >>>>>>>>>>> should just be retried until the rebalance is complete
> >>>>>>>>>>> * StateStoreMigratedException: the store handle is
> >>>>>>>>>>> now invalid, so you need to re-discover the instance
> >>>>>>>>>>> and get a new handle on that instance. In other words,
> >>>>>>>>>>> the query itself may be valid, but the particular method
> >>>>>>>>>>> invocation on this particular instance has encountered
> >>>>>>>>>>> a fatal exception.
> >>>>>>>>>>>
> >>>>>>>>>>> Fatal:
> >>>>>>>>>>> * UnknownStateStoreException: this is truly fatal. No amount
> >>>>>>>>>>> of retrying or re-discovering is going to get you a handle
> >> on a
> >>>>>>>>>>> store that doesn't exist in the cluster.
> >>>>>>>>>>> * StateStoreNotAvailableException: this is actually
> recoverable,
> >>>>>>>>>>> since the store might exist in the cluster, but isn't
> >> available
> >>>>>> on
> >>>>>>>>>>> this particular instance (which is shut down or whatever).
> >>>>>>>>>>>
> >>>>>>>>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> >>>>>>>>>>> with omitting the categorization and just having 5 subclasses
> >>>>>>>>>>> of InvalidStateStoreException. Each of them would tell you
> >>>>>>>>>>> how to handle them, and it's not too many to really
> >>>>>>>>>>> understand and handle each one.
> >>>>>>>>>>>
> >>>>>>>>>>> If you really want to have a middle tier, I'd recommend:
> >>>>>>>>>>> * RetryableStateStoreException: the exact same call
> >>>>>>>>>>> should be repeated.
> >>>>>>>>>>> * RecoverableStateStoreException: the store handle
> >>>>>>>>>>> should be discarded and the caller should re-discover
> >>>>>>>>>>> the location of the store and repeat the query on the
> >>>>>>>>>>> correct instance.
> >>>>>>>>>>> * FatalStateStoreException: the query/request is totally
> >>>>>>>>>>> invalid and will never succeed.
> >>>>>>>>>>>
> >>>>>>>>>>> However, attempting to categorize the proposed exceptions
> >>>>>>>>>>> reveals even problems with this categorization:
> >>>>>>>>>>> Retriable:
> >>>>>>>>>>> * StreamsRebalancingException
> >>>>>>>>>>> Recoverable:
> >>>>>>>>>>> * StateStoreMigratedException
> >>>>>>>>>>> * StreamsNotRunningException
> >>>>>>>>>>> Fatal:
> >>>>>>>>>>> * UnknownStateStoreException
> >>>>>>>>>>>
> >>>>>>>>>>> But StreamsNotStartedException is strange... It means that
> >>>>>>>>>>> one code path got a handle on a specific KafkaStreams object
> >>>>>>>>>>> instance and sent it a query before another code path
> >>>>>>>>>>> invoked the start() method on the exact same object instance.
> >>>>>>>>>>> It seems like the most likely scenario is that whoever wrote
> >>>>>>>>>>> the program just forgot to call start() before querying, in
> >>>>>>>>>>> which case, retrying isn't going to help, and a fatal exception
> >>>>>>>>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
> >>>>>>>>>>> experience" problem, and making it fatal would be more
> >>>>>>>>>>> helpful. Even in a production context, there's no reason not
> >>>>>>>>>>> to sequence your application startup such that you don't
> >>>>>>>>>>> accept queries until after Streams is started. Thus, I guess
> >>>>>>>>>>> I'd categorize it under "fatal".
> >>>>>>>>>>>
> >>>>>>>>>>> Regardless of whether you make it fatal or retriable, you'd
> >>>>>>>>>>> still have a whole category with only one exception in it,
> >>>>>>>>>>> and the other two categories only have two exceptions.
> >>>>>>>>>>> Plus, as you pointed out in the KIP, you can't get all
> >>>>>>>>>>> exceptions in all cases anyway:
> >>>>>>>>>>> * store() can only throw NotStarted, NotRunning,
> >>>>>>>>>>> and Unknown
> >>>>>>>>>>> * actual store queries can only throw Rebalancing,
> >>>>>>>>>>> Migrated, and NotRunning
> >>>>>>>>>>>
> >>>>>>>>>>> Thus, in practice also, there are exactly three categories
> >>>>>>>>>>> and also exactly three exception types. It doesn't seem
> >>>>>>>>>>> like there's a great advantage to the categories here. To
> >>>>>>>>>>> avoid the categorization problem and also to clarify what
> >>>>>>>>>>> exceptions can actually be thrown in different circumstances,
> >>>>>>>>>>> it seems like we should just:
> >>>>>>>>>>> * get rid of the middle tier and make all the exceptions
> >>>>>>>>>>> extend InvalidStateStoreException
> >>>>>>>>>>> * drop StateStoreNotAvailableException in favor of
> >>>>>>>>>>> StreamsNotRunningException
> >>>>>>>>>>> * clearly document on all public methods which exceptions
> >>>>>>>>>>> need to be handled
> >>>>>>>>>>>
> >>>>>>>>>>> How do you feel about this?
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> -John
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> >>>>>>>>>>>> Thanks for KIP Vito.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Overall the KIP LGTM, but I'd have to agree with others on
> >> merging
> >>>>>>>> the
> >>>>>>>>>>>> `StreamsNotRunningException` and
> >> `StateStoreNotAvailableException`
> >>>>>>>>>>> classes.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN
> ||
> >>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could
> distinguish
> >>>>>>>> when to
> >>>>>>>>>>>> use the different
> >>>>>>>>>>>> exceptions. Maybe a good middle ground would be to have a
> >> detailed
> >>>>>>>>>>>> exception message.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The KIP freeze is close, so I think if we can agree on this,
> we
> >> can
> >>>>>>>>>>> wrap up
> >>>>>>>>>>>> the voting soon.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Bill
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
> >>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It's still unclear to me what the advantage is, to have both
> >>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING
> /
> >>>>>>>> ERROR`
> >>>>>>>>>>>>> and thus, for a user point of view, why does it matter if the
> >>>>>>>> store is
> >>>>>>>>>>>>> closed on not? I don't understand why/how this information
> >> would
> >>>>>>>> be
> >>>>>>>>>>>>> useful? Do you have a concrete example in mind how a user
> would
> >>>>>>>> react
> >>>>>>>>>>>>> differently to both exceptions?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it
> seems
> >>>>>>>> best
> >>>>>>>>>>> to
> >>>>>>>>>>>>> actually do this on a per-query basis, ie, have an overload
> >>>>>>>>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that
> allow
> >> to
> >>>>>>>>>>>>> _disable_ the exception and opt-in to query a active store
> >> during
> >>>>>>>>>>>>> recovery. However, as KIP-535 actually introduces this change
> >> in
> >>>>>>>>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
> >>>>>>>> should be
> >>>>>>>>>>>>> updated. I'll follow up on the other KIP thread to raise this
> >>>>>>>> point.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
> >>>>>>>>>>>>>> Hi, Matthias & Vinoth,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> What is still unclear to me is, what we gain by having both
> >>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>> `StateStoreNotAvailableException`. Both
> >>>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
> >>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to
> know
> >>>>>>>> if the
> >>>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
> >> Maybe
> >>>>>>>> I
> >>>>>>>>>>> miss
> >>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Yes, both `StreamsNotRunningException` and
> >>>>>>>>>>>>>> `StateStoreNotAvailableException` are fatal exception.
> >>>>>>>>>>>>>> But `StateStoreNotAvailableException` is fatal exception
> about
> >>>>>>>> state
> >>>>>>>>>>>>> store
> >>>>>>>>>>>>>> related.
> >>>>>>>>>>>>>> I think it would be helpful that if user need to distinguish
> >>>>>>>> these
> >>>>>>>>>>> two
> >>>>>>>>>>>>>> different case to handle it.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm not very sure, does that make sense?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
> >>>>>>>> vinoth@apache.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> +1 on merging `StreamsNotRunningException` and
> >>>>>>>>>>>>>>> `StateStoreNotAvailableException`, both exceptions are
> fatal
> >>>>>>>>>>> anyway. IMO
> >>>>>>>>>>>>>>> its best to have these exceptions be about the state store
> >>>>>>>> (and not
> >>>>>>>>>>>>> streams
> >>>>>>>>>>>>>>> state), to easier understanding.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Additionally, KIP-535 allows for querying of state stores
> in
> >>>>>>>>>>> rebalancing
> >>>>>>>>>>>>>>> state. So do we need the StreamsRebalancingException?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
> >>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>> Sorry that I dropped the ball on this...
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free
> to
> >>>>>>>> start
> >>>>>>>>>>> a
> >>>>>>>>>>>>> VOTE
> >>>>>>>>>>>>>>>> thread.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> What is still unclear to me is, what we gain by having
> both
> >>>>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>>>> `StateStoreNotAvailableException`.
> >>>>>>>>>>>>> Both
> >>>>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
> >>>>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to
> know
> >>>>>>>> if
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
> >>>>>>>> Maybe I
> >>>>>>>>>>> miss
> >>>>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> >>>>>>>>>>>>>>>>> Sorry for the late reply, thanks for the review.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A
> store
> >>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
> >> resume
> >>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> >>>>>>>> store
> >>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense
> during
> >>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thank you point this, already updated.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Why do we need to distinguish between
> >>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by
> various
> >>>>>>>>>>> reasons, I
> >>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>> it would be helpful that the
> >>>>>>>>>>>>>>>>> user can distinguish whether it is caused by the state
> >> store
> >>>>>>>>>>> closed.
> >>>>>>>>>>>>>>>>> (Maybe I am wrong...)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
> instance
> >>>>>>>> and
> >>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer
> to
> >>>>>>>> the
> >>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>> because that is the level of granularity in which we
> >>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>> IQ atm.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Totally agree. Do you mean the naming of state store
> >>>>>>>> exceptions?
> >>>>>>>>>>>>>>>>> I don't have special reason to distinguish these two.
> >>>>>>>>>>>>>>>>> Your suggestion look more reasonable for the exception
> >>>>>>>> naming.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add
> that a
> >>>>>>>> user
> >>>>>>>>>>> need
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the
> store
> >>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That
> is
> >> a
> >>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >>>>>>>> "blind"
> >>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
> >>>>>>>> instance
> >>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
> >>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The KIP already updated, please take a look. :)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> >>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Any update on this KIP?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >>>>>>>>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us
> quite
> >>>>>>>> busy.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A
> >> store
> >>>>>>>>>>> might be
> >>>>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
> >> resume
> >>>>>>>>>>> back to
> >>>>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an
> old
> >>>>>>>> store
> >>>>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense
> during
> >>>>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Why do we need to distinguish between
> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
> >>>>>>>> instance and
> >>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer
> >> to
> >>>>>>>> the
> >>>>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>>>> because that is the level of granularity in which we
> >>>>>>>>>>> enable/disable
> >>>>>>>>>>>>>>> IQ
> >>>>>>>>>>>>>>>>>> atm.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add
> >> that a
> >>>>>>>>>>> user
> >>>>>>>>>>>>>>> need to
> >>>>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the
> >> store
> >>>>>>>>>>> handle is
> >>>>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That
> >> is a
> >>>>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >>>>>>>> "blind"
> >>>>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
> >>>>>>>> instance
> >>>>>>>>>>>>> after
> >>>>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
> >>>>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >>>>>>>>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >> <https://shorturl.at/CDNT9>
> >>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Please use the following instead:
> >>>>>>>> https://shorturl.at/bkKQU
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
> >>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
> >>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> For the case that corresponding thread crashes after
> we
> >>>>>>>>>>> handed out
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> store handle. We may throw
> >>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
> >>>>>>>>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread
> is
> >>>>>>>> not
> >>>>>>>>>>>>>>> running(
> >>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
> >>>>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I
> think
> >>>>>>>> we
> >>>>>>>>>>> do not
> >>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
> >>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I understand your point. I rename
> >>>>>>>>>>>>>>> `StreamThreadNotRunningException` to
> >>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> About check unknown state store names:
> >>>>>>>>>>>>>>>>>>>>> Thank you for the hint. I add a new type
> >>>>>>>>>>>>>>> `UnknownStateStoreException`
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>> this case.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Also, we should still have fatal exception
> >>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
> >>>>>>>> remove
> >>>>>>>>>>> it?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thank you point this, already add it again.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> The KIP already updated, please take a look.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks Vito.
I am also ok with either name. Just a personal slight preference, but
not a important.
-Matthias
On 1/21/20 6:52 PM, Vito Jeng wrote:
> Thanks Matthias.
>
> The KIP is about InvalidStateStoreException.
> I pick `StateStoreNotAvailableException` because it may be more intuitive
> than `StreamsNotRunningException`.
>
> No matter which one picked, it's good to me.
>
> ---
> Vito
>
>
> On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Thanks for updating the KIP!
>>
>> One last comment/question: you kept `StateStoreNotAvailableException` in
>> favor of `StreamsNotRunningException` (to merge both as suggested).
>>
>> I am wondering, if it might be better to keep
>> `StreamsNotRunningException` instead of
>> `StateStoreNotAvailableException`, because this exception is thrown if
>> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
>>
>>
>>
>> -Matthias
>>
>> On 1/17/20 9:56 PM, John Roesler wrote:
>>> Thanks, Vito. I've just cast my vote.
>>> -John
>>>
>>> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>>>> Hi, folks,
>>>>
>>>> Just update the KIP, please take a look.
>>>>
>>>> Thanks!
>>>>
>>>> ---
>>>> Vito
>>>>
>>>>
>>>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>>>>
>>>>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
>>>>> I got a lot out of the discussion.
>>>>>
>>>>> I would like to update KIP-216 base on John's suggestion to remove the
>>>>> category.
>>>>>
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>>
>>>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <matthias@confluent.io
>>>
>>>>> wrote:
>>>>>
>>>>>>> Nevertheless, if we omit the categorization, it’s moot.
>>>>>>
>>>>>> Ack.
>>>>>>
>>>>>> I am fine to remove the middle tier. As John pointed out, it might be
>>>>>> weird to have only one concrete exception type per category. We can
>> also
>>>>>> explain in detail how to handle each exception in their JavaDocs.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>>>>>>> Vito,
>>>>>>>
>>>>>>> Thanks for the updates, the KIP LGTM.
>>>>>>>
>>>>>>> -Bill
>>>>>>>
>>>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org>
>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Vito,
>>>>>>>>
>>>>>>>> Haha, your archive game is on point!
>>>>>>>>
>>>>>>>> What Matthias said in that email is essentially what I figured was
>> the
>>>>>>>> rationale. It makes sense, but the point I was making is that this
>>>>>> really
>>>>>>>> doesn’t seem like a good way to structure a production app. On the
>>>>>> other
>>>>>>>> hand, considering the exception fatal has a good chance of avoiding
>> a
>>>>>>>> frustrating debug session if you just forgot to call start.
>>>>>>>>
>>>>>>>> Nevertheless, if we omit the categorization, it’s moot.
>>>>>>>>
>>>>>>>> It would be easy to add a categorization layer later if we want it,
>> but
>>>>>>>> not very easy to change it if we get it wrong.
>>>>>>>>
>>>>>>>> Thanks for your consideration!
>>>>>>>> -John
>>>>>>>>
>>>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
>>>>>>>>> Hi John,
>>>>>>>>>
>>>>>>>>> About `StreamsNotStartedException is strange` --
>>>>>>>>> The original idea came from Matthias, two years ago. :)
>>>>>>>>> You can reference here:
>>>>>>>>>
>>>>>>>>
>>>>>>
>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>>>>>>>>>
>>>>>>>>> About omitting the categorization --
>>>>>>>>> It looks reasonable. I'm fine with omitting the categorization but
>> not
>>>>>>>> very
>>>>>>>>> sure it is a good choice.
>>>>>>>>> Does any other folks provide opinion?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi, folks,
>>>>>>>>>
>>>>>>>>> Just update the KIP-216, please take a look.
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Vito
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi, folks,
>>>>>>>>>>
>>>>>>>>>> Thank you suggestion, really appreciate it. :)
>>>>>>>>>> I understand your concern. I'll merge StreamsNotRunningException
>> and
>>>>>>>>>> StateStoreNotAvailableException.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vvcephei@apache.org
>>>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Vito,
>>>>>>>>>>>
>>>>>>>>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>>>>>>>>>>> Hopefully, we can close it out soon.
>>>>>>>>>>>
>>>>>>>>>>> I agree we can drop StreamsNotRunningException in favor of
>>>>>>>>>>> just StateStoreNotAvailableException.
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately, I have some higher-level concerns. The value
>>>>>>>>>>> of these exceptions is that they tell you how to handle the
>>>>>>>>>>> various situations that can arise while querying a distributed
>>>>>>>>>>> data store.
>>>>>>>>>>>
>>>>>>>>>>> Ideally, as a caller, I should be able to just catch "retriable"
>> or
>>>>>>>>>>> "fatal" and handle them appropriately. Otherwise, there's no
>>>>>>>>>>> point in having categories, and we should just have all the
>>>>>>>>>>> exceptions extend InvalidStateStoreException.
>>>>>>>>>>>
>>>>>>>>>>> Presently, it's not possible to tell from just the
>>>>>>>>>>> "retriable"/"fatal" distinction what to do. You can tell
>>>>>>>>>>> from the descriptions of the various exceptions. E.g.:
>>>>>>>>>>>
>>>>>>>>>>> Retriable:
>>>>>>>>>>> * StreamsRebalancingException: the exact same call
>>>>>>>>>>> should just be retried until the rebalance is complete
>>>>>>>>>>> * StateStoreMigratedException: the store handle is
>>>>>>>>>>> now invalid, so you need to re-discover the instance
>>>>>>>>>>> and get a new handle on that instance. In other words,
>>>>>>>>>>> the query itself may be valid, but the particular method
>>>>>>>>>>> invocation on this particular instance has encountered
>>>>>>>>>>> a fatal exception.
>>>>>>>>>>>
>>>>>>>>>>> Fatal:
>>>>>>>>>>> * UnknownStateStoreException: this is truly fatal. No amount
>>>>>>>>>>> of retrying or re-discovering is going to get you a handle
>> on a
>>>>>>>>>>> store that doesn't exist in the cluster.
>>>>>>>>>>> * StateStoreNotAvailableException: this is actually recoverable,
>>>>>>>>>>> since the store might exist in the cluster, but isn't
>> available
>>>>>> on
>>>>>>>>>>> this particular instance (which is shut down or whatever).
>>>>>>>>>>>
>>>>>>>>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
>>>>>>>>>>> with omitting the categorization and just having 5 subclasses
>>>>>>>>>>> of InvalidStateStoreException. Each of them would tell you
>>>>>>>>>>> how to handle them, and it's not too many to really
>>>>>>>>>>> understand and handle each one.
>>>>>>>>>>>
>>>>>>>>>>> If you really want to have a middle tier, I'd recommend:
>>>>>>>>>>> * RetryableStateStoreException: the exact same call
>>>>>>>>>>> should be repeated.
>>>>>>>>>>> * RecoverableStateStoreException: the store handle
>>>>>>>>>>> should be discarded and the caller should re-discover
>>>>>>>>>>> the location of the store and repeat the query on the
>>>>>>>>>>> correct instance.
>>>>>>>>>>> * FatalStateStoreException: the query/request is totally
>>>>>>>>>>> invalid and will never succeed.
>>>>>>>>>>>
>>>>>>>>>>> However, attempting to categorize the proposed exceptions
>>>>>>>>>>> reveals even problems with this categorization:
>>>>>>>>>>> Retriable:
>>>>>>>>>>> * StreamsRebalancingException
>>>>>>>>>>> Recoverable:
>>>>>>>>>>> * StateStoreMigratedException
>>>>>>>>>>> * StreamsNotRunningException
>>>>>>>>>>> Fatal:
>>>>>>>>>>> * UnknownStateStoreException
>>>>>>>>>>>
>>>>>>>>>>> But StreamsNotStartedException is strange... It means that
>>>>>>>>>>> one code path got a handle on a specific KafkaStreams object
>>>>>>>>>>> instance and sent it a query before another code path
>>>>>>>>>>> invoked the start() method on the exact same object instance.
>>>>>>>>>>> It seems like the most likely scenario is that whoever wrote
>>>>>>>>>>> the program just forgot to call start() before querying, in
>>>>>>>>>>> which case, retrying isn't going to help, and a fatal exception
>>>>>>>>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
>>>>>>>>>>> experience" problem, and making it fatal would be more
>>>>>>>>>>> helpful. Even in a production context, there's no reason not
>>>>>>>>>>> to sequence your application startup such that you don't
>>>>>>>>>>> accept queries until after Streams is started. Thus, I guess
>>>>>>>>>>> I'd categorize it under "fatal".
>>>>>>>>>>>
>>>>>>>>>>> Regardless of whether you make it fatal or retriable, you'd
>>>>>>>>>>> still have a whole category with only one exception in it,
>>>>>>>>>>> and the other two categories only have two exceptions.
>>>>>>>>>>> Plus, as you pointed out in the KIP, you can't get all
>>>>>>>>>>> exceptions in all cases anyway:
>>>>>>>>>>> * store() can only throw NotStarted, NotRunning,
>>>>>>>>>>> and Unknown
>>>>>>>>>>> * actual store queries can only throw Rebalancing,
>>>>>>>>>>> Migrated, and NotRunning
>>>>>>>>>>>
>>>>>>>>>>> Thus, in practice also, there are exactly three categories
>>>>>>>>>>> and also exactly three exception types. It doesn't seem
>>>>>>>>>>> like there's a great advantage to the categories here. To
>>>>>>>>>>> avoid the categorization problem and also to clarify what
>>>>>>>>>>> exceptions can actually be thrown in different circumstances,
>>>>>>>>>>> it seems like we should just:
>>>>>>>>>>> * get rid of the middle tier and make all the exceptions
>>>>>>>>>>> extend InvalidStateStoreException
>>>>>>>>>>> * drop StateStoreNotAvailableException in favor of
>>>>>>>>>>> StreamsNotRunningException
>>>>>>>>>>> * clearly document on all public methods which exceptions
>>>>>>>>>>> need to be handled
>>>>>>>>>>>
>>>>>>>>>>> How do you feel about this?
>>>>>>>>>>> Thanks,
>>>>>>>>>>> -John
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
>>>>>>>>>>>> Thanks for KIP Vito.
>>>>>>>>>>>>
>>>>>>>>>>>> Overall the KIP LGTM, but I'd have to agree with others on
>> merging
>>>>>>>> the
>>>>>>>>>>>> `StreamsNotRunningException` and
>> `StateStoreNotAvailableException`
>>>>>>>>>>> classes.
>>>>>>>>>>>>
>>>>>>>>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
>>>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
>>>>>>>> when to
>>>>>>>>>>>> use the different
>>>>>>>>>>>> exceptions. Maybe a good middle ground would be to have a
>> detailed
>>>>>>>>>>>> exception message.
>>>>>>>>>>>>
>>>>>>>>>>>> The KIP freeze is close, so I think if we can agree on this, we
>> can
>>>>>>>>>>> wrap up
>>>>>>>>>>>> the voting soon.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Bill
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's still unclear to me what the advantage is, to have both
>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>>>
>>>>>>>>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
>>>>>>>> ERROR`
>>>>>>>>>>>>> and thus, for a user point of view, why does it matter if the
>>>>>>>> store is
>>>>>>>>>>>>> closed on not? I don't understand why/how this information
>> would
>>>>>>>> be
>>>>>>>>>>>>> useful? Do you have a concrete example in mind how a user would
>>>>>>>> react
>>>>>>>>>>>>> differently to both exceptions?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
>>>>>>>> best
>>>>>>>>>>> to
>>>>>>>>>>>>> actually do this on a per-query basis, ie, have an overload
>>>>>>>>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow
>> to
>>>>>>>>>>>>> _disable_ the exception and opt-in to query a active store
>> during
>>>>>>>>>>>>> recovery. However, as KIP-535 actually introduces this change
>> in
>>>>>>>>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
>>>>>>>> should be
>>>>>>>>>>>>> updated. I'll follow up on the other KIP thread to raise this
>>>>>>>> point.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
>>>>>>>>>>>>>> Hi, Matthias & Vinoth,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> What is still unclear to me is, what we gain by having both
>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>> `StateStoreNotAvailableException`. Both
>>>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>>>>>>>> if the
>>>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
>> Maybe
>>>>>>>> I
>>>>>>>>>>> miss
>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, both `StreamsNotRunningException` and
>>>>>>>>>>>>>> `StateStoreNotAvailableException` are fatal exception.
>>>>>>>>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
>>>>>>>> state
>>>>>>>>>>>>> store
>>>>>>>>>>>>>> related.
>>>>>>>>>>>>>> I think it would be helpful that if user need to distinguish
>>>>>>>> these
>>>>>>>>>>> two
>>>>>>>>>>>>>> different case to handle it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm not very sure, does that make sense?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
>>>>>>>> vinoth@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +1 on merging `StreamsNotRunningException` and
>>>>>>>>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
>>>>>>>>>>> anyway. IMO
>>>>>>>>>>>>>>> its best to have these exceptions be about the state store
>>>>>>>> (and not
>>>>>>>>>>>>> streams
>>>>>>>>>>>>>>> state), to easier understanding.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
>>>>>>>>>>> rebalancing
>>>>>>>>>>>>>>> state. So do we need the StreamsRebalancingException?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Sorry that I dropped the ball on this...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
>>>>>>>> start
>>>>>>>>>>> a
>>>>>>>>>>>>> VOTE
>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What is still unclear to me is, what we gain by having both
>>>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>>>> `StateStoreNotAvailableException`.
>>>>>>>>>>>>> Both
>>>>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
>>>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>>>>>>>> if
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
>>>>>>>> Maybe I
>>>>>>>>>>> miss
>>>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>>>>>>>>>>>>>>>>> Sorry for the late reply, thanks for the review.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
>> resume
>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>>>>>>>> store
>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you point this, already updated.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Why do we need to distinguish between
>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
>>>>>>>>>>> reasons, I
>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>> it would be helpful that the
>>>>>>>>>>>>>>>>> user can distinguish whether it is caused by the state
>> store
>>>>>>>>>>> closed.
>>>>>>>>>>>>>>>>> (Maybe I am wrong...)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
>>>>>>>> and
>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
>>>>>>>> the
>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>> because that is the level of granularity in which we
>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>> IQ atm.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Totally agree. Do you mean the naming of state store
>>>>>>>> exceptions?
>>>>>>>>>>>>>>>>> I don't have special reason to distinguish these two.
>>>>>>>>>>>>>>>>> Your suggestion look more reasonable for the exception
>>>>>>>> naming.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
>>>>>>>> user
>>>>>>>>>>> need
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is
>> a
>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>>>>>>>> "blind"
>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
>>>>>>>> instance
>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Nice, it's great! Thank you.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The KIP already updated, please take a look. :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Any update on this KIP?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>>>>>>>>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
>>>>>>>> busy.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A
>> store
>>>>>>>>>>> might be
>>>>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
>> resume
>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>>>>>>>> store
>>>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>>>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>>>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Why do we need to distinguish between
>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
>>>>>>>> instance and
>>>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer
>> to
>>>>>>>> the
>>>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>>>> because that is the level of granularity in which we
>>>>>>>>>>> enable/disable
>>>>>>>>>>>>>>> IQ
>>>>>>>>>>>>>>>>>> atm.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add
>> that a
>>>>>>>>>>> user
>>>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the
>> store
>>>>>>>>>>> handle is
>>>>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That
>> is a
>>>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>>>>>>>> "blind"
>>>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
>>>>>>>> instance
>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
>>>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>>>>>>>>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9>
>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please use the following instead:
>>>>>>>> https://shorturl.at/bkKQU
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
>>>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
>>>>>>>>>>> handed out
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> store handle. We may throw
>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
>>>>>>>>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
>>>>>>>> not
>>>>>>>>>>>>>>> running(
>>>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
>>>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
>>>>>>>> we
>>>>>>>>>>> do not
>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I understand your point. I rename
>>>>>>>>>>>>>>> `StreamThreadNotRunningException` to
>>>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> About check unknown state store names:
>>>>>>>>>>>>>>>>>>>>> Thank you for the hint. I add a new type
>>>>>>>>>>>>>>> `UnknownStateStoreException`
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>> this case.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Also, we should still have fatal exception
>>>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
>>>>>>>> remove
>>>>>>>>>>> it?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you point this, already add it again.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The KIP already updated, please take a look.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Thanks Matthias.
The KIP is about InvalidStateStoreException.
I pick `StateStoreNotAvailableException` because it may be more intuitive
than `StreamsNotRunningException`.
No matter which one picked, it's good to me.
---
Vito
On Wed, Jan 22, 2020 at 7:44 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for updating the KIP!
>
> One last comment/question: you kept `StateStoreNotAvailableException` in
> favor of `StreamsNotRunningException` (to merge both as suggested).
>
> I am wondering, if it might be better to keep
> `StreamsNotRunningException` instead of
> `StateStoreNotAvailableException`, because this exception is thrown if
> Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
>
>
>
> -Matthias
>
> On 1/17/20 9:56 PM, John Roesler wrote:
> > Thanks, Vito. I've just cast my vote.
> > -John
> >
> > On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> >> Hi, folks,
> >>
> >> Just update the KIP, please take a look.
> >>
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> >>
> >>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> >>> I got a lot out of the discussion.
> >>>
> >>> I would like to update KIP-216 base on John's suggestion to remove the
> >>> category.
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <matthias@confluent.io
> >
> >>> wrote:
> >>>
> >>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>
> >>>> Ack.
> >>>>
> >>>> I am fine to remove the middle tier. As John pointed out, it might be
> >>>> weird to have only one concrete exception type per category. We can
> also
> >>>> explain in detail how to handle each exception in their JavaDocs.
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >>>>> Vito,
> >>>>>
> >>>>> Thanks for the updates, the KIP LGTM.
> >>>>>
> >>>>> -Bill
> >>>>>
> >>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org>
> >>>> wrote:
> >>>>>
> >>>>>> Hi Vito,
> >>>>>>
> >>>>>> Haha, your archive game is on point!
> >>>>>>
> >>>>>> What Matthias said in that email is essentially what I figured was
> the
> >>>>>> rationale. It makes sense, but the point I was making is that this
> >>>> really
> >>>>>> doesn’t seem like a good way to structure a production app. On the
> >>>> other
> >>>>>> hand, considering the exception fatal has a good chance of avoiding
> a
> >>>>>> frustrating debug session if you just forgot to call start.
> >>>>>>
> >>>>>> Nevertheless, if we omit the categorization, it’s moot.
> >>>>>>
> >>>>>> It would be easy to add a categorization layer later if we want it,
> but
> >>>>>> not very easy to change it if we get it wrong.
> >>>>>>
> >>>>>> Thanks for your consideration!
> >>>>>> -John
> >>>>>>
> >>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> >>>>>>> Hi John,
> >>>>>>>
> >>>>>>> About `StreamsNotStartedException is strange` --
> >>>>>>> The original idea came from Matthias, two years ago. :)
> >>>>>>> You can reference here:
> >>>>>>>
> >>>>>>
> >>>>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> >>>>>>>
> >>>>>>> About omitting the categorization --
> >>>>>>> It looks reasonable. I'm fine with omitting the categorization but
> not
> >>>>>> very
> >>>>>>> sure it is a good choice.
> >>>>>>> Does any other folks provide opinion?
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi, folks,
> >>>>>>>
> >>>>>>> Just update the KIP-216, please take a look.
> >>>>>>>
> >>>>>>> ---
> >>>>>>> Vito
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw>
> >>>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Hi, folks,
> >>>>>>>>
> >>>>>>>> Thank you suggestion, really appreciate it. :)
> >>>>>>>> I understand your concern. I'll merge StreamsNotRunningException
> and
> >>>>>>>> StateStoreNotAvailableException.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Vito
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vvcephei@apache.org
> >
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hey Vito,
> >>>>>>>>>
> >>>>>>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
> >>>>>>>>> Hopefully, we can close it out soon.
> >>>>>>>>>
> >>>>>>>>> I agree we can drop StreamsNotRunningException in favor of
> >>>>>>>>> just StateStoreNotAvailableException.
> >>>>>>>>>
> >>>>>>>>> Unfortunately, I have some higher-level concerns. The value
> >>>>>>>>> of these exceptions is that they tell you how to handle the
> >>>>>>>>> various situations that can arise while querying a distributed
> >>>>>>>>> data store.
> >>>>>>>>>
> >>>>>>>>> Ideally, as a caller, I should be able to just catch "retriable"
> or
> >>>>>>>>> "fatal" and handle them appropriately. Otherwise, there's no
> >>>>>>>>> point in having categories, and we should just have all the
> >>>>>>>>> exceptions extend InvalidStateStoreException.
> >>>>>>>>>
> >>>>>>>>> Presently, it's not possible to tell from just the
> >>>>>>>>> "retriable"/"fatal" distinction what to do. You can tell
> >>>>>>>>> from the descriptions of the various exceptions. E.g.:
> >>>>>>>>>
> >>>>>>>>> Retriable:
> >>>>>>>>> * StreamsRebalancingException: the exact same call
> >>>>>>>>> should just be retried until the rebalance is complete
> >>>>>>>>> * StateStoreMigratedException: the store handle is
> >>>>>>>>> now invalid, so you need to re-discover the instance
> >>>>>>>>> and get a new handle on that instance. In other words,
> >>>>>>>>> the query itself may be valid, but the particular method
> >>>>>>>>> invocation on this particular instance has encountered
> >>>>>>>>> a fatal exception.
> >>>>>>>>>
> >>>>>>>>> Fatal:
> >>>>>>>>> * UnknownStateStoreException: this is truly fatal. No amount
> >>>>>>>>> of retrying or re-discovering is going to get you a handle
> on a
> >>>>>>>>> store that doesn't exist in the cluster.
> >>>>>>>>> * StateStoreNotAvailableException: this is actually recoverable,
> >>>>>>>>> since the store might exist in the cluster, but isn't
> available
> >>>> on
> >>>>>>>>> this particular instance (which is shut down or whatever).
> >>>>>>>>>
> >>>>>>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> >>>>>>>>> with omitting the categorization and just having 5 subclasses
> >>>>>>>>> of InvalidStateStoreException. Each of them would tell you
> >>>>>>>>> how to handle them, and it's not too many to really
> >>>>>>>>> understand and handle each one.
> >>>>>>>>>
> >>>>>>>>> If you really want to have a middle tier, I'd recommend:
> >>>>>>>>> * RetryableStateStoreException: the exact same call
> >>>>>>>>> should be repeated.
> >>>>>>>>> * RecoverableStateStoreException: the store handle
> >>>>>>>>> should be discarded and the caller should re-discover
> >>>>>>>>> the location of the store and repeat the query on the
> >>>>>>>>> correct instance.
> >>>>>>>>> * FatalStateStoreException: the query/request is totally
> >>>>>>>>> invalid and will never succeed.
> >>>>>>>>>
> >>>>>>>>> However, attempting to categorize the proposed exceptions
> >>>>>>>>> reveals even problems with this categorization:
> >>>>>>>>> Retriable:
> >>>>>>>>> * StreamsRebalancingException
> >>>>>>>>> Recoverable:
> >>>>>>>>> * StateStoreMigratedException
> >>>>>>>>> * StreamsNotRunningException
> >>>>>>>>> Fatal:
> >>>>>>>>> * UnknownStateStoreException
> >>>>>>>>>
> >>>>>>>>> But StreamsNotStartedException is strange... It means that
> >>>>>>>>> one code path got a handle on a specific KafkaStreams object
> >>>>>>>>> instance and sent it a query before another code path
> >>>>>>>>> invoked the start() method on the exact same object instance.
> >>>>>>>>> It seems like the most likely scenario is that whoever wrote
> >>>>>>>>> the program just forgot to call start() before querying, in
> >>>>>>>>> which case, retrying isn't going to help, and a fatal exception
> >>>>>>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
> >>>>>>>>> experience" problem, and making it fatal would be more
> >>>>>>>>> helpful. Even in a production context, there's no reason not
> >>>>>>>>> to sequence your application startup such that you don't
> >>>>>>>>> accept queries until after Streams is started. Thus, I guess
> >>>>>>>>> I'd categorize it under "fatal".
> >>>>>>>>>
> >>>>>>>>> Regardless of whether you make it fatal or retriable, you'd
> >>>>>>>>> still have a whole category with only one exception in it,
> >>>>>>>>> and the other two categories only have two exceptions.
> >>>>>>>>> Plus, as you pointed out in the KIP, you can't get all
> >>>>>>>>> exceptions in all cases anyway:
> >>>>>>>>> * store() can only throw NotStarted, NotRunning,
> >>>>>>>>> and Unknown
> >>>>>>>>> * actual store queries can only throw Rebalancing,
> >>>>>>>>> Migrated, and NotRunning
> >>>>>>>>>
> >>>>>>>>> Thus, in practice also, there are exactly three categories
> >>>>>>>>> and also exactly three exception types. It doesn't seem
> >>>>>>>>> like there's a great advantage to the categories here. To
> >>>>>>>>> avoid the categorization problem and also to clarify what
> >>>>>>>>> exceptions can actually be thrown in different circumstances,
> >>>>>>>>> it seems like we should just:
> >>>>>>>>> * get rid of the middle tier and make all the exceptions
> >>>>>>>>> extend InvalidStateStoreException
> >>>>>>>>> * drop StateStoreNotAvailableException in favor of
> >>>>>>>>> StreamsNotRunningException
> >>>>>>>>> * clearly document on all public methods which exceptions
> >>>>>>>>> need to be handled
> >>>>>>>>>
> >>>>>>>>> How do you feel about this?
> >>>>>>>>> Thanks,
> >>>>>>>>> -John
> >>>>>>>>>
> >>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> >>>>>>>>>> Thanks for KIP Vito.
> >>>>>>>>>>
> >>>>>>>>>> Overall the KIP LGTM, but I'd have to agree with others on
> merging
> >>>>>> the
> >>>>>>>>>> `StreamsNotRunningException` and
> `StateStoreNotAvailableException`
> >>>>>>>>> classes.
> >>>>>>>>>>
> >>>>>>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> >>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
> >>>>>> when to
> >>>>>>>>>> use the different
> >>>>>>>>>> exceptions. Maybe a good middle ground would be to have a
> detailed
> >>>>>>>>>> exception message.
> >>>>>>>>>>
> >>>>>>>>>> The KIP freeze is close, so I think if we can agree on this, we
> can
> >>>>>>>>> wrap up
> >>>>>>>>>> the voting soon.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Bill
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
> >>>>>> matthias@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Vito,
> >>>>>>>>>>>
> >>>>>>>>>>> It's still unclear to me what the advantage is, to have both
> >>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>> `StateStoreNotAvailableException`?
> >>>>>>>>>>>
> >>>>>>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
> >>>>>> ERROR`
> >>>>>>>>>>> and thus, for a user point of view, why does it matter if the
> >>>>>> store is
> >>>>>>>>>>> closed on not? I don't understand why/how this information
> would
> >>>>>> be
> >>>>>>>>>>> useful? Do you have a concrete example in mind how a user would
> >>>>>> react
> >>>>>>>>>>> differently to both exceptions?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
> >>>>>> best
> >>>>>>>>> to
> >>>>>>>>>>> actually do this on a per-query basis, ie, have an overload
> >>>>>>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow
> to
> >>>>>>>>>>> _disable_ the exception and opt-in to query a active store
> during
> >>>>>>>>>>> recovery. However, as KIP-535 actually introduces this change
> in
> >>>>>>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
> >>>>>> should be
> >>>>>>>>>>> updated. I'll follow up on the other KIP thread to raise this
> >>>>>> point.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -Matthias
> >>>>>>>>>>>
> >>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
> >>>>>>>>>>>> Hi, Matthias & Vinoth,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the feedback.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> What is still unclear to me is, what we gain by having both
> >>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>> `StateStoreNotAvailableException`. Both
> >>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
> >>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> >>>>>> if the
> >>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
> Maybe
> >>>>>> I
> >>>>>>>>> miss
> >>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes, both `StreamsNotRunningException` and
> >>>>>>>>>>>> `StateStoreNotAvailableException` are fatal exception.
> >>>>>>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
> >>>>>> state
> >>>>>>>>>>> store
> >>>>>>>>>>>> related.
> >>>>>>>>>>>> I think it would be helpful that if user need to distinguish
> >>>>>> these
> >>>>>>>>> two
> >>>>>>>>>>>> different case to handle it.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'm not very sure, does that make sense?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
> >>>>>> vinoth@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> +1 on merging `StreamsNotRunningException` and
> >>>>>>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
> >>>>>>>>> anyway. IMO
> >>>>>>>>>>>>> its best to have these exceptions be about the state store
> >>>>>> (and not
> >>>>>>>>>>> streams
> >>>>>>>>>>>>> state), to easier understanding.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
> >>>>>>>>> rebalancing
> >>>>>>>>>>>>> state. So do we need the StreamsRebalancingException?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
> >>>>>> matthias@confluent.io>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> Sorry that I dropped the ball on this...
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
> >>>>>> start
> >>>>>>>>> a
> >>>>>>>>>>> VOTE
> >>>>>>>>>>>>>> thread.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What is still unclear to me is, what we gain by having both
> >>>>>>>>>>>>>> `StreamsNotRunningException` and
> >>>>>>>>> `StateStoreNotAvailableException`.
> >>>>>>>>>>> Both
> >>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
> >>>>>>>>> PENDING_SHUTDOWN /
> >>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> >>>>>> if
> >>>>>>>>> the
> >>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
> >>>>>> Maybe I
> >>>>>>>>> miss
> >>>>>>>>>>>>>> something thought?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> >>>>>>>>>>>>>>> Sorry for the late reply, thanks for the review.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
> >>>>>>>>> might be
> >>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
> resume
> >>>>>>>>> back to
> >>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> >>>>>> store
> >>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> >>>>>>>>> `RUNNING`
> >>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thank you point this, already updated.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Why do we need to distinguish between
> >>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
> >>>>>>>>> reasons, I
> >>>>>>>>>>>>> think
> >>>>>>>>>>>>>>> it would be helpful that the
> >>>>>>>>>>>>>>> user can distinguish whether it is caused by the state
> store
> >>>>>>>>> closed.
> >>>>>>>>>>>>>>> (Maybe I am wrong...)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
> >>>>>> and
> >>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
> >>>>>> the
> >>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>> because that is the level of granularity in which we
> >>>>>>>>> enable/disable
> >>>>>>>>>>>>> IQ atm.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Totally agree. Do you mean the naming of state store
> >>>>>> exceptions?
> >>>>>>>>>>>>>>> I don't have special reason to distinguish these two.
> >>>>>>>>>>>>>>> Your suggestion look more reasonable for the exception
> >>>>>> naming.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
> >>>>>> user
> >>>>>>>>> need
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
> >>>>>>>>> handle is
> >>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is
> a
> >>>>>>>>>>> difference
> >>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >>>>>> "blind"
> >>>>>>>>> retries
> >>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
> >>>>>> instance
> >>>>>>>>> after
> >>>>>>>>>>>>>>>> rebalancing finishes, or changes to
> >>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> store was migrated away during rebalancing).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Nice, it's great! Thank you.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The KIP already updated, please take a look. :)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> >>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Any update on this KIP?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >>>>>>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
> >>>>>> busy.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A
> store
> >>>>>>>>> might be
> >>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might
> resume
> >>>>>>>>> back to
> >>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> >>>>>> store
> >>>>>>>>>>>>> handle.
> >>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> >>>>>>>>> `RUNNING`
> >>>>>>>>>>>>>>>> state.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Why do we need to distinguish between
> >>>>>>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
> >>>>>> instance and
> >>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer
> to
> >>>>>> the
> >>>>>>>>>>>>> instance,
> >>>>>>>>>>>>>>>>> because that is the level of granularity in which we
> >>>>>>>>> enable/disable
> >>>>>>>>>>>>> IQ
> >>>>>>>>>>>>>>>> atm.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add
> that a
> >>>>>>>>> user
> >>>>>>>>>>>>> need to
> >>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the
> store
> >>>>>>>>> handle is
> >>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That
> is a
> >>>>>>>>>>>>> difference
> >>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >>>>>> "blind"
> >>>>>>>>>>> retries
> >>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
> >>>>>> instance
> >>>>>>>>>>> after
> >>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
> >>>>>>>>> `StateStoreMigratedException` if
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >>>>>>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >>>> <https://shorturl.at/CDNT9>
> >>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Please use the following instead:
> >>>>>> https://shorturl.at/bkKQU
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
> >>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks, Matthias!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
> >>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
> >>>>>>>>> handed out
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> store handle. We may throw
> >>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>> or
> >>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
> >>>>>>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
> >>>>>> not
> >>>>>>>>>>>>> running(
> >>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
> >>>>>>>>> `StateStoreMigratedException`
> >>>>>>>>>>>>> when
> >>>>>>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
> >>>>>> we
> >>>>>>>>> do not
> >>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
> >>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I understand your point. I rename
> >>>>>>>>>>>>> `StreamThreadNotRunningException` to
> >>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> About check unknown state store names:
> >>>>>>>>>>>>>>>>>>> Thank you for the hint. I add a new type
> >>>>>>>>>>>>> `UnknownStateStoreException`
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> this case.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Also, we should still have fatal exception
> >>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
> >>>>>> remove
> >>>>>>>>> it?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thank you point this, already add it again.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> The KIP already updated, please take a look.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks for updating the KIP!
One last comment/question: you kept `StateStoreNotAvailableException` in
favor of `StreamsNotRunningException` (to merge both as suggested).
I am wondering, if it might be better to keep
`StreamsNotRunningException` instead of
`StateStoreNotAvailableException`, because this exception is thrown if
Streams is in state PENDING_SHUTDOWN / NOT_RUNNING / ERROR ?
-Matthias
On 1/17/20 9:56 PM, John Roesler wrote:
> Thanks, Vito. I've just cast my vote.
> -John
>
> On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
>> Hi, folks,
>>
>> Just update the KIP, please take a look.
>>
>> Thanks!
>>
>> ---
>> Vito
>>
>>
>> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>>
>>> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
>>> I got a lot out of the discussion.
>>>
>>> I would like to update KIP-216 base on John's suggestion to remove the
>>> category.
>>>
>>>
>>> ---
>>> Vito
>>>
>>>
>>> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>>> Nevertheless, if we omit the categorization, it’s moot.
>>>>
>>>> Ack.
>>>>
>>>> I am fine to remove the middle tier. As John pointed out, it might be
>>>> weird to have only one concrete exception type per category. We can also
>>>> explain in detail how to handle each exception in their JavaDocs.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>>>>> Vito,
>>>>>
>>>>> Thanks for the updates, the KIP LGTM.
>>>>>
>>>>> -Bill
>>>>>
>>>>> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Hi Vito,
>>>>>>
>>>>>> Haha, your archive game is on point!
>>>>>>
>>>>>> What Matthias said in that email is essentially what I figured was the
>>>>>> rationale. It makes sense, but the point I was making is that this
>>>> really
>>>>>> doesn’t seem like a good way to structure a production app. On the
>>>> other
>>>>>> hand, considering the exception fatal has a good chance of avoiding a
>>>>>> frustrating debug session if you just forgot to call start.
>>>>>>
>>>>>> Nevertheless, if we omit the categorization, it’s moot.
>>>>>>
>>>>>> It would be easy to add a categorization layer later if we want it, but
>>>>>> not very easy to change it if we get it wrong.
>>>>>>
>>>>>> Thanks for your consideration!
>>>>>> -John
>>>>>>
>>>>>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
>>>>>>> Hi John,
>>>>>>>
>>>>>>> About `StreamsNotStartedException is strange` --
>>>>>>> The original idea came from Matthias, two years ago. :)
>>>>>>> You can reference here:
>>>>>>>
>>>>>>
>>>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>>>>>>>
>>>>>>> About omitting the categorization --
>>>>>>> It looks reasonable. I'm fine with omitting the categorization but not
>>>>>> very
>>>>>>> sure it is a good choice.
>>>>>>> Does any other folks provide opinion?
>>>>>>>
>>>>>>>
>>>>>>> Hi, folks,
>>>>>>>
>>>>>>> Just update the KIP-216, please take a look.
>>>>>>>
>>>>>>> ---
>>>>>>> Vito
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw>
>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi, folks,
>>>>>>>>
>>>>>>>> Thank you suggestion, really appreciate it. :)
>>>>>>>> I understand your concern. I'll merge StreamsNotRunningException and
>>>>>>>> StateStoreNotAvailableException.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hey Vito,
>>>>>>>>>
>>>>>>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>>>>>>>>> Hopefully, we can close it out soon.
>>>>>>>>>
>>>>>>>>> I agree we can drop StreamsNotRunningException in favor of
>>>>>>>>> just StateStoreNotAvailableException.
>>>>>>>>>
>>>>>>>>> Unfortunately, I have some higher-level concerns. The value
>>>>>>>>> of these exceptions is that they tell you how to handle the
>>>>>>>>> various situations that can arise while querying a distributed
>>>>>>>>> data store.
>>>>>>>>>
>>>>>>>>> Ideally, as a caller, I should be able to just catch "retriable" or
>>>>>>>>> "fatal" and handle them appropriately. Otherwise, there's no
>>>>>>>>> point in having categories, and we should just have all the
>>>>>>>>> exceptions extend InvalidStateStoreException.
>>>>>>>>>
>>>>>>>>> Presently, it's not possible to tell from just the
>>>>>>>>> "retriable"/"fatal" distinction what to do. You can tell
>>>>>>>>> from the descriptions of the various exceptions. E.g.:
>>>>>>>>>
>>>>>>>>> Retriable:
>>>>>>>>> * StreamsRebalancingException: the exact same call
>>>>>>>>> should just be retried until the rebalance is complete
>>>>>>>>> * StateStoreMigratedException: the store handle is
>>>>>>>>> now invalid, so you need to re-discover the instance
>>>>>>>>> and get a new handle on that instance. In other words,
>>>>>>>>> the query itself may be valid, but the particular method
>>>>>>>>> invocation on this particular instance has encountered
>>>>>>>>> a fatal exception.
>>>>>>>>>
>>>>>>>>> Fatal:
>>>>>>>>> * UnknownStateStoreException: this is truly fatal. No amount
>>>>>>>>> of retrying or re-discovering is going to get you a handle on a
>>>>>>>>> store that doesn't exist in the cluster.
>>>>>>>>> * StateStoreNotAvailableException: this is actually recoverable,
>>>>>>>>> since the store might exist in the cluster, but isn't available
>>>> on
>>>>>>>>> this particular instance (which is shut down or whatever).
>>>>>>>>>
>>>>>>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
>>>>>>>>> with omitting the categorization and just having 5 subclasses
>>>>>>>>> of InvalidStateStoreException. Each of them would tell you
>>>>>>>>> how to handle them, and it's not too many to really
>>>>>>>>> understand and handle each one.
>>>>>>>>>
>>>>>>>>> If you really want to have a middle tier, I'd recommend:
>>>>>>>>> * RetryableStateStoreException: the exact same call
>>>>>>>>> should be repeated.
>>>>>>>>> * RecoverableStateStoreException: the store handle
>>>>>>>>> should be discarded and the caller should re-discover
>>>>>>>>> the location of the store and repeat the query on the
>>>>>>>>> correct instance.
>>>>>>>>> * FatalStateStoreException: the query/request is totally
>>>>>>>>> invalid and will never succeed.
>>>>>>>>>
>>>>>>>>> However, attempting to categorize the proposed exceptions
>>>>>>>>> reveals even problems with this categorization:
>>>>>>>>> Retriable:
>>>>>>>>> * StreamsRebalancingException
>>>>>>>>> Recoverable:
>>>>>>>>> * StateStoreMigratedException
>>>>>>>>> * StreamsNotRunningException
>>>>>>>>> Fatal:
>>>>>>>>> * UnknownStateStoreException
>>>>>>>>>
>>>>>>>>> But StreamsNotStartedException is strange... It means that
>>>>>>>>> one code path got a handle on a specific KafkaStreams object
>>>>>>>>> instance and sent it a query before another code path
>>>>>>>>> invoked the start() method on the exact same object instance.
>>>>>>>>> It seems like the most likely scenario is that whoever wrote
>>>>>>>>> the program just forgot to call start() before querying, in
>>>>>>>>> which case, retrying isn't going to help, and a fatal exception
>>>>>>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
>>>>>>>>> experience" problem, and making it fatal would be more
>>>>>>>>> helpful. Even in a production context, there's no reason not
>>>>>>>>> to sequence your application startup such that you don't
>>>>>>>>> accept queries until after Streams is started. Thus, I guess
>>>>>>>>> I'd categorize it under "fatal".
>>>>>>>>>
>>>>>>>>> Regardless of whether you make it fatal or retriable, you'd
>>>>>>>>> still have a whole category with only one exception in it,
>>>>>>>>> and the other two categories only have two exceptions.
>>>>>>>>> Plus, as you pointed out in the KIP, you can't get all
>>>>>>>>> exceptions in all cases anyway:
>>>>>>>>> * store() can only throw NotStarted, NotRunning,
>>>>>>>>> and Unknown
>>>>>>>>> * actual store queries can only throw Rebalancing,
>>>>>>>>> Migrated, and NotRunning
>>>>>>>>>
>>>>>>>>> Thus, in practice also, there are exactly three categories
>>>>>>>>> and also exactly three exception types. It doesn't seem
>>>>>>>>> like there's a great advantage to the categories here. To
>>>>>>>>> avoid the categorization problem and also to clarify what
>>>>>>>>> exceptions can actually be thrown in different circumstances,
>>>>>>>>> it seems like we should just:
>>>>>>>>> * get rid of the middle tier and make all the exceptions
>>>>>>>>> extend InvalidStateStoreException
>>>>>>>>> * drop StateStoreNotAvailableException in favor of
>>>>>>>>> StreamsNotRunningException
>>>>>>>>> * clearly document on all public methods which exceptions
>>>>>>>>> need to be handled
>>>>>>>>>
>>>>>>>>> How do you feel about this?
>>>>>>>>> Thanks,
>>>>>>>>> -John
>>>>>>>>>
>>>>>>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
>>>>>>>>>> Thanks for KIP Vito.
>>>>>>>>>>
>>>>>>>>>> Overall the KIP LGTM, but I'd have to agree with others on merging
>>>>>> the
>>>>>>>>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`
>>>>>>>>> classes.
>>>>>>>>>>
>>>>>>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
>>>>>>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
>>>>>> when to
>>>>>>>>>> use the different
>>>>>>>>>> exceptions. Maybe a good middle ground would be to have a detailed
>>>>>>>>>> exception message.
>>>>>>>>>>
>>>>>>>>>> The KIP freeze is close, so I think if we can agree on this, we can
>>>>>>>>> wrap up
>>>>>>>>>> the voting soon.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bill
>>>>>>>>>>
>>>>>>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
>>>>>> matthias@confluent.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Vito,
>>>>>>>>>>>
>>>>>>>>>>> It's still unclear to me what the advantage is, to have both
>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>> `StateStoreNotAvailableException`?
>>>>>>>>>>>
>>>>>>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
>>>>>> ERROR`
>>>>>>>>>>> and thus, for a user point of view, why does it matter if the
>>>>>> store is
>>>>>>>>>>> closed on not? I don't understand why/how this information would
>>>>>> be
>>>>>>>>>>> useful? Do you have a concrete example in mind how a user would
>>>>>> react
>>>>>>>>>>> differently to both exceptions?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
>>>>>> best
>>>>>>>>> to
>>>>>>>>>>> actually do this on a per-query basis, ie, have an overload
>>>>>>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow to
>>>>>>>>>>> _disable_ the exception and opt-in to query a active store during
>>>>>>>>>>> recovery. However, as KIP-535 actually introduces this change in
>>>>>>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
>>>>>> should be
>>>>>>>>>>> updated. I'll follow up on the other KIP thread to raise this
>>>>>> point.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
>>>>>>>>>>>> Hi, Matthias & Vinoth,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the feedback.
>>>>>>>>>>>>
>>>>>>>>>>>>> What is still unclear to me is, what we gain by having both
>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>> `StateStoreNotAvailableException`. Both
>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>>>>>> if the
>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway? Maybe
>>>>>> I
>>>>>>>>> miss
>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, both `StreamsNotRunningException` and
>>>>>>>>>>>> `StateStoreNotAvailableException` are fatal exception.
>>>>>>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
>>>>>> state
>>>>>>>>>>> store
>>>>>>>>>>>> related.
>>>>>>>>>>>> I think it would be helpful that if user need to distinguish
>>>>>> these
>>>>>>>>> two
>>>>>>>>>>>> different case to handle it.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not very sure, does that make sense?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Vito
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
>>>>>> vinoth@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> +1 on merging `StreamsNotRunningException` and
>>>>>>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
>>>>>>>>> anyway. IMO
>>>>>>>>>>>>> its best to have these exceptions be about the state store
>>>>>> (and not
>>>>>>>>>>> streams
>>>>>>>>>>>>> state), to easier understanding.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
>>>>>>>>> rebalancing
>>>>>>>>>>>>> state. So do we need the StreamsRebalancingException?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
>>>>>> matthias@confluent.io>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Sorry that I dropped the ball on this...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
>>>>>> start
>>>>>>>>> a
>>>>>>>>>>> VOTE
>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What is still unclear to me is, what we gain by having both
>>>>>>>>>>>>>> `StreamsNotRunningException` and
>>>>>>>>> `StateStoreNotAvailableException`.
>>>>>>>>>>> Both
>>>>>>>>>>>>>> exception are thrown when KafkaStreams is in state
>>>>>>>>> PENDING_SHUTDOWN /
>>>>>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>>>>>> if
>>>>>>>>> the
>>>>>>>>>>>>>> state store is closed on not -- I can't query it anyway?
>>>>>> Maybe I
>>>>>>>>> miss
>>>>>>>>>>>>>> something thought?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>>>>>>>>>>>>>>> Sorry for the late reply, thanks for the review.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
>>>>>>>>> might be
>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
>>>>>>>>> back to
>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>>>>>> store
>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>>>>>>>>> `RUNNING`
>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you point this, already updated.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Why do we need to distinguish between
>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
>>>>>>>>> reasons, I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>>> it would be helpful that the
>>>>>>>>>>>>>>> user can distinguish whether it is caused by the state store
>>>>>>>>> closed.
>>>>>>>>>>>>>>> (Maybe I am wrong...)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
>>>>>> and
>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
>>>>>> the
>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>> because that is the level of granularity in which we
>>>>>>>>> enable/disable
>>>>>>>>>>>>> IQ atm.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Totally agree. Do you mean the naming of state store
>>>>>> exceptions?
>>>>>>>>>>>>>>> I don't have special reason to distinguish these two.
>>>>>>>>>>>>>>> Your suggestion look more reasonable for the exception
>>>>>> naming.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
>>>>>> user
>>>>>>>>> need
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
>>>>>>>>> handle is
>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
>>>>>>>>>>> difference
>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>>>>>> "blind"
>>>>>>>>> retries
>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
>>>>>> instance
>>>>>>>>> after
>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Nice, it's great! Thank you.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The KIP already updated, please take a look. :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Any update on this KIP?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>>>>>>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
>>>>>> busy.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> About `StateStoreMigratedException`:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
>>>>>>>>> might be
>>>>>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
>>>>>>>>> back to
>>>>>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>>>>>> store
>>>>>>>>>>>>> handle.
>>>>>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>>>>>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>>>>>>>>> `RUNNING`
>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Why do we need to distinguish between
>>>>>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
>>>>>> instance and
>>>>>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
>>>>>> the
>>>>>>>>>>>>> instance,
>>>>>>>>>>>>>>>>> because that is the level of granularity in which we
>>>>>>>>> enable/disable
>>>>>>>>>>>>> IQ
>>>>>>>>>>>>>>>> atm.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
>>>>>>>>> user
>>>>>>>>>>>>> need to
>>>>>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
>>>>>>>>> handle is
>>>>>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
>>>>>>>>>>>>> difference
>>>>>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>>>>>> "blind"
>>>>>>>>>>> retries
>>>>>>>>>>>>>>>>> that either resolve (if the store is still on the same
>>>>>> instance
>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> rebalancing finishes, or changes to
>>>>>>>>> `StateStoreMigratedException` if
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> store was migrated away during rebalancing).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>>>>>>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>>>> <https://shorturl.at/CDNT9>
>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please use the following instead:
>>>>>> https://shorturl.at/bkKQU
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks, Matthias!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
>>>>>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>>>>>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
>>>>>>>>> handed out
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> store handle. We may throw
>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>> or
>>>>>>>>>>>>>>>>>>> `StateStoreMigratedException`.
>>>>>>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
>>>>>> not
>>>>>>>>>>>>> running(
>>>>>>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
>>>>>>>>> `StateStoreMigratedException`
>>>>>>>>>>>>> when
>>>>>>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
>>>>>> we
>>>>>>>>> do not
>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I understand your point. I rename
>>>>>>>>>>>>> `StreamThreadNotRunningException` to
>>>>>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> About check unknown state store names:
>>>>>>>>>>>>>>>>>>> Thank you for the hint. I add a new type
>>>>>>>>>>>>> `UnknownStateStoreException`
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> this case.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Also, we should still have fatal exception
>>>>>>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
>>>>>> remove
>>>>>>>>> it?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you point this, already add it again.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The KIP already updated, please take a look.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors
Posted by John Roesler <vv...@apache.org>.
Thanks, Vito. I've just cast my vote.
-John
On Fri, Jan 17, 2020, at 21:32, Vito Jeng wrote:
> Hi, folks,
>
> Just update the KIP, please take a look.
>
> Thanks!
>
> ---
> Vito
>
>
> On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>
> > Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> > I got a lot out of the discussion.
> >
> > I would like to update KIP-216 base on John's suggestion to remove the
> > category.
> >
> >
> > ---
> > Vito
> >
> >
> > On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> > Nevertheless, if we omit the categorization, it’s moot.
> >>
> >> Ack.
> >>
> >> I am fine to remove the middle tier. As John pointed out, it might be
> >> weird to have only one concrete exception type per category. We can also
> >> explain in detail how to handle each exception in their JavaDocs.
> >>
> >>
> >> -Matthias
> >>
> >> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> >> > Vito,
> >> >
> >> > Thanks for the updates, the KIP LGTM.
> >> >
> >> > -Bill
> >> >
> >> > On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org>
> >> wrote:
> >> >
> >> >> Hi Vito,
> >> >>
> >> >> Haha, your archive game is on point!
> >> >>
> >> >> What Matthias said in that email is essentially what I figured was the
> >> >> rationale. It makes sense, but the point I was making is that this
> >> really
> >> >> doesn’t seem like a good way to structure a production app. On the
> >> other
> >> >> hand, considering the exception fatal has a good chance of avoiding a
> >> >> frustrating debug session if you just forgot to call start.
> >> >>
> >> >> Nevertheless, if we omit the categorization, it’s moot.
> >> >>
> >> >> It would be easy to add a categorization layer later if we want it, but
> >> >> not very easy to change it if we get it wrong.
> >> >>
> >> >> Thanks for your consideration!
> >> >> -John
> >> >>
> >> >> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> >> >>> Hi John,
> >> >>>
> >> >>> About `StreamsNotStartedException is strange` --
> >> >>> The original idea came from Matthias, two years ago. :)
> >> >>> You can reference here:
> >> >>>
> >> >>
> >> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> >> >>>
> >> >>> About omitting the categorization --
> >> >>> It looks reasonable. I'm fine with omitting the categorization but not
> >> >> very
> >> >>> sure it is a good choice.
> >> >>> Does any other folks provide opinion?
> >> >>>
> >> >>>
> >> >>> Hi, folks,
> >> >>>
> >> >>> Just update the KIP-216, please take a look.
> >> >>>
> >> >>> ---
> >> >>> Vito
> >> >>>
> >> >>>
> >> >>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw>
> >> wrote:
> >> >>>
> >> >>>>
> >> >>>> Hi, folks,
> >> >>>>
> >> >>>> Thank you suggestion, really appreciate it. :)
> >> >>>> I understand your concern. I'll merge StreamsNotRunningException and
> >> >>>> StateStoreNotAvailableException.
> >> >>>>
> >> >>>>
> >> >>>> ---
> >> >>>> Vito
> >> >>>>
> >> >>>>
> >> >>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org>
> >> >> wrote:
> >> >>>>
> >> >>>>> Hey Vito,
> >> >>>>>
> >> >>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
> >> >>>>> Hopefully, we can close it out soon.
> >> >>>>>
> >> >>>>> I agree we can drop StreamsNotRunningException in favor of
> >> >>>>> just StateStoreNotAvailableException.
> >> >>>>>
> >> >>>>> Unfortunately, I have some higher-level concerns. The value
> >> >>>>> of these exceptions is that they tell you how to handle the
> >> >>>>> various situations that can arise while querying a distributed
> >> >>>>> data store.
> >> >>>>>
> >> >>>>> Ideally, as a caller, I should be able to just catch "retriable" or
> >> >>>>> "fatal" and handle them appropriately. Otherwise, there's no
> >> >>>>> point in having categories, and we should just have all the
> >> >>>>> exceptions extend InvalidStateStoreException.
> >> >>>>>
> >> >>>>> Presently, it's not possible to tell from just the
> >> >>>>> "retriable"/"fatal" distinction what to do. You can tell
> >> >>>>> from the descriptions of the various exceptions. E.g.:
> >> >>>>>
> >> >>>>> Retriable:
> >> >>>>> * StreamsRebalancingException: the exact same call
> >> >>>>> should just be retried until the rebalance is complete
> >> >>>>> * StateStoreMigratedException: the store handle is
> >> >>>>> now invalid, so you need to re-discover the instance
> >> >>>>> and get a new handle on that instance. In other words,
> >> >>>>> the query itself may be valid, but the particular method
> >> >>>>> invocation on this particular instance has encountered
> >> >>>>> a fatal exception.
> >> >>>>>
> >> >>>>> Fatal:
> >> >>>>> * UnknownStateStoreException: this is truly fatal. No amount
> >> >>>>> of retrying or re-discovering is going to get you a handle on a
> >> >>>>> store that doesn't exist in the cluster.
> >> >>>>> * StateStoreNotAvailableException: this is actually recoverable,
> >> >>>>> since the store might exist in the cluster, but isn't available
> >> on
> >> >>>>> this particular instance (which is shut down or whatever).
> >> >>>>>
> >> >>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> >> >>>>> with omitting the categorization and just having 5 subclasses
> >> >>>>> of InvalidStateStoreException. Each of them would tell you
> >> >>>>> how to handle them, and it's not too many to really
> >> >>>>> understand and handle each one.
> >> >>>>>
> >> >>>>> If you really want to have a middle tier, I'd recommend:
> >> >>>>> * RetryableStateStoreException: the exact same call
> >> >>>>> should be repeated.
> >> >>>>> * RecoverableStateStoreException: the store handle
> >> >>>>> should be discarded and the caller should re-discover
> >> >>>>> the location of the store and repeat the query on the
> >> >>>>> correct instance.
> >> >>>>> * FatalStateStoreException: the query/request is totally
> >> >>>>> invalid and will never succeed.
> >> >>>>>
> >> >>>>> However, attempting to categorize the proposed exceptions
> >> >>>>> reveals even problems with this categorization:
> >> >>>>> Retriable:
> >> >>>>> * StreamsRebalancingException
> >> >>>>> Recoverable:
> >> >>>>> * StateStoreMigratedException
> >> >>>>> * StreamsNotRunningException
> >> >>>>> Fatal:
> >> >>>>> * UnknownStateStoreException
> >> >>>>>
> >> >>>>> But StreamsNotStartedException is strange... It means that
> >> >>>>> one code path got a handle on a specific KafkaStreams object
> >> >>>>> instance and sent it a query before another code path
> >> >>>>> invoked the start() method on the exact same object instance.
> >> >>>>> It seems like the most likely scenario is that whoever wrote
> >> >>>>> the program just forgot to call start() before querying, in
> >> >>>>> which case, retrying isn't going to help, and a fatal exception
> >> >>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
> >> >>>>> experience" problem, and making it fatal would be more
> >> >>>>> helpful. Even in a production context, there's no reason not
> >> >>>>> to sequence your application startup such that you don't
> >> >>>>> accept queries until after Streams is started. Thus, I guess
> >> >>>>> I'd categorize it under "fatal".
> >> >>>>>
> >> >>>>> Regardless of whether you make it fatal or retriable, you'd
> >> >>>>> still have a whole category with only one exception in it,
> >> >>>>> and the other two categories only have two exceptions.
> >> >>>>> Plus, as you pointed out in the KIP, you can't get all
> >> >>>>> exceptions in all cases anyway:
> >> >>>>> * store() can only throw NotStarted, NotRunning,
> >> >>>>> and Unknown
> >> >>>>> * actual store queries can only throw Rebalancing,
> >> >>>>> Migrated, and NotRunning
> >> >>>>>
> >> >>>>> Thus, in practice also, there are exactly three categories
> >> >>>>> and also exactly three exception types. It doesn't seem
> >> >>>>> like there's a great advantage to the categories here. To
> >> >>>>> avoid the categorization problem and also to clarify what
> >> >>>>> exceptions can actually be thrown in different circumstances,
> >> >>>>> it seems like we should just:
> >> >>>>> * get rid of the middle tier and make all the exceptions
> >> >>>>> extend InvalidStateStoreException
> >> >>>>> * drop StateStoreNotAvailableException in favor of
> >> >>>>> StreamsNotRunningException
> >> >>>>> * clearly document on all public methods which exceptions
> >> >>>>> need to be handled
> >> >>>>>
> >> >>>>> How do you feel about this?
> >> >>>>> Thanks,
> >> >>>>> -John
> >> >>>>>
> >> >>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> >> >>>>>> Thanks for KIP Vito.
> >> >>>>>>
> >> >>>>>> Overall the KIP LGTM, but I'd have to agree with others on merging
> >> >> the
> >> >>>>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`
> >> >>>>> classes.
> >> >>>>>>
> >> >>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> >> >>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
> >> >> when to
> >> >>>>>> use the different
> >> >>>>>> exceptions. Maybe a good middle ground would be to have a detailed
> >> >>>>>> exception message.
> >> >>>>>>
> >> >>>>>> The KIP freeze is close, so I think if we can agree on this, we can
> >> >>>>> wrap up
> >> >>>>>> the voting soon.
> >> >>>>>>
> >> >>>>>> Thanks,
> >> >>>>>> Bill
> >> >>>>>>
> >> >>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
> >> >> matthias@confluent.io>
> >> >>>>>> wrote:
> >> >>>>>>
> >> >>>>>>> Vito,
> >> >>>>>>>
> >> >>>>>>> It's still unclear to me what the advantage is, to have both
> >> >>>>>>> `StreamsNotRunningException` and
> >> >> `StateStoreNotAvailableException`?
> >> >>>>>>>
> >> >>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
> >> >> ERROR`
> >> >>>>>>> and thus, for a user point of view, why does it matter if the
> >> >> store is
> >> >>>>>>> closed on not? I don't understand why/how this information would
> >> >> be
> >> >>>>>>> useful? Do you have a concrete example in mind how a user would
> >> >> react
> >> >>>>>>> differently to both exceptions?
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
> >> >> best
> >> >>>>> to
> >> >>>>>>> actually do this on a per-query basis, ie, have an overload
> >> >>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow to
> >> >>>>>>> _disable_ the exception and opt-in to query a active store during
> >> >>>>>>> recovery. However, as KIP-535 actually introduces this change in
> >> >>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
> >> >> should be
> >> >>>>>>> updated. I'll follow up on the other KIP thread to raise this
> >> >> point.
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>> -Matthias
> >> >>>>>>>
> >> >>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
> >> >>>>>>>> Hi, Matthias & Vinoth,
> >> >>>>>>>>
> >> >>>>>>>> Thanks for the feedback.
> >> >>>>>>>>
> >> >>>>>>>>> What is still unclear to me is, what we gain by having both
> >> >>>>>>>>> `StreamsNotRunningException` and
> >> >>>>> `StateStoreNotAvailableException`. Both
> >> >>>>>>>>> exception are thrown when KafkaStreams is in state
> >> >>>>> PENDING_SHUTDOWN /
> >> >>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> >> >> if the
> >> >>>>>>>>> state store is closed on not -- I can't query it anyway? Maybe
> >> >> I
> >> >>>>> miss
> >> >>>>>>>>> something thought?
> >> >>>>>>>>
> >> >>>>>>>> Yes, both `StreamsNotRunningException` and
> >> >>>>>>>> `StateStoreNotAvailableException` are fatal exception.
> >> >>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
> >> >> state
> >> >>>>>>> store
> >> >>>>>>>> related.
> >> >>>>>>>> I think it would be helpful that if user need to distinguish
> >> >> these
> >> >>>>> two
> >> >>>>>>>> different case to handle it.
> >> >>>>>>>>
> >> >>>>>>>> I'm not very sure, does that make sense?
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> ---
> >> >>>>>>>> Vito
> >> >>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
> >> >> vinoth@apache.org>
> >> >>>>>>> wrote:
> >> >>>>>>>>
> >> >>>>>>>>> +1 on merging `StreamsNotRunningException` and
> >> >>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
> >> >>>>> anyway. IMO
> >> >>>>>>>>> its best to have these exceptions be about the state store
> >> >> (and not
> >> >>>>>>> streams
> >> >>>>>>>>> state), to easier understanding.
> >> >>>>>>>>>
> >> >>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
> >> >>>>> rebalancing
> >> >>>>>>>>> state. So do we need the StreamsRebalancingException?
> >> >>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
> >> >> matthias@confluent.io>
> >> >>>>>>> wrote:
> >> >>>>>>>>>> Sorry that I dropped the ball on this...
> >> >>>>>>>>>>
> >> >>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
> >> >> start
> >> >>>>> a
> >> >>>>>>> VOTE
> >> >>>>>>>>>> thread.
> >> >>>>>>>>>>
> >> >>>>>>>>>> What is still unclear to me is, what we gain by having both
> >> >>>>>>>>>> `StreamsNotRunningException` and
> >> >>>>> `StateStoreNotAvailableException`.
> >> >>>>>>> Both
> >> >>>>>>>>>> exception are thrown when KafkaStreams is in state
> >> >>>>> PENDING_SHUTDOWN /
> >> >>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> >> >> if
> >> >>>>> the
> >> >>>>>>>>>> state store is closed on not -- I can't query it anyway?
> >> >> Maybe I
> >> >>>>> miss
> >> >>>>>>>>>> something thought?
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> -Matthias
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> >> >>>>>>>>>>> Sorry for the late reply, thanks for the review.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> About `StateStoreMigratedException`:
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
> >> >>>>> might be
> >> >>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
> >> >>>>> back to
> >> >>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> >> >> store
> >> >>>>>>> handle.
> >> >>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >> >>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >> >>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> >> >>>>> `RUNNING`
> >> >>>>>>>>> state.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Thank you point this, already updated.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Why do we need to distinguish between
> >> >>>>>>> `KafkaStreamsNotRunningException`
> >> >>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
> >> >>>>> reasons, I
> >> >>>>>>>>> think
> >> >>>>>>>>>>> it would be helpful that the
> >> >>>>>>>>>>> user can distinguish whether it is caused by the state store
> >> >>>>> closed.
> >> >>>>>>>>>>> (Maybe I am wrong...)
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
> >> >> and
> >> >>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
> >> >> the
> >> >>>>>>>>> instance,
> >> >>>>>>>>>>>> because that is the level of granularity in which we
> >> >>>>> enable/disable
> >> >>>>>>>>> IQ atm.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Totally agree. Do you mean the naming of state store
> >> >> exceptions?
> >> >>>>>>>>>>> I don't have special reason to distinguish these two.
> >> >>>>>>>>>>> Your suggestion look more reasonable for the exception
> >> >> naming.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
> >> >> user
> >> >>>>> need
> >> >>>>>>>>> to
> >> >>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
> >> >>>>> handle is
> >> >>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
> >> >>>>>>> difference
> >> >>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >> >> "blind"
> >> >>>>> retries
> >> >>>>>>>>>>>> that either resolve (if the store is still on the same
> >> >> instance
> >> >>>>> after
> >> >>>>>>>>>>>> rebalancing finishes, or changes to
> >> >>>>> `StateStoreMigratedException` if
> >> >>>>>>>>> the
> >> >>>>>>>>>>>> store was migrated away during rebalancing).
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Nice, it's great! Thank you.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> The KIP already updated, please take a look. :)
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> >> >>>>>>> matthias@confluent.io
> >> >>>>>>>>>>
> >> >>>>>>>>>>> wrote:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Any update on this KIP?
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >> >>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
> >> >> busy.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> About `StateStoreMigratedException`:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
> >> >>>>> might be
> >> >>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
> >> >>>>> back to
> >> >>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> >> >> store
> >> >>>>>>>>> handle.
> >> >>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >> >>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >> >>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> >> >>>>> `RUNNING`
> >> >>>>>>>>>>>> state.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Why do we need to distinguish between
> >> >>>>>>>>> `KafkaStreamsNotRunningException`
> >> >>>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
> >> >> instance and
> >> >>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
> >> >> the
> >> >>>>>>>>> instance,
> >> >>>>>>>>>>>>> because that is the level of granularity in which we
> >> >>>>> enable/disable
> >> >>>>>>>>> IQ
> >> >>>>>>>>>>>> atm.
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
> >> >>>>> user
> >> >>>>>>>>> need to
> >> >>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
> >> >>>>> handle is
> >> >>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
> >> >>>>>>>>> difference
> >> >>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >> >> "blind"
> >> >>>>>>> retries
> >> >>>>>>>>>>>>> that either resolve (if the store is still on the same
> >> >> instance
> >> >>>>>>> after
> >> >>>>>>>>>>>>> rebalancing finishes, or changes to
> >> >>>>> `StateStoreMigratedException` if
> >> >>>>>>>>> the
> >> >>>>>>>>>>>>> store was migrated away during rebalancing).
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> -Matthias
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >> >>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> >> <https://shorturl.at/CDNT9>
> >> >> <https://shorturl.at/CDNT9>
> >> >>>>> <https://shorturl.at/CDNT9>
> >> >>>>>>> <https://shorturl.at/CDNT9>
> >> >>>>>>>>> <https://shorturl.at/CDNT9>
> >> >>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >> >>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Please use the following instead:
> >> >> https://shorturl.at/bkKQU
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> ---
> >> >>>>>>>>>>>>>> Vito
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
> >> >>>>> vito@is-land.com.tw>
> >> >>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Thanks, Matthias!
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
> >> >>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >> >>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
> >> >>>>> handed out
> >> >>>>>>>>> the
> >> >>>>>>>>>>>>>>> store handle. We may throw
> >> >> `KafkaStreamsNotRunningException`
> >> >>>>> or
> >> >>>>>>>>>>>>>>> `StateStoreMigratedException`.
> >> >>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> >> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
> >> >> not
> >> >>>>>>>>> running(
> >> >>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
> >> >>>>> `StateStoreMigratedException`
> >> >>>>>>>>> when
> >> >>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
> >> >> we
> >> >>>>> do not
> >> >>>>>>>>> need
> >> >>>>>>>>>>>> to
> >> >>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
> >> >>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> I understand your point. I rename
> >> >>>>>>>>> `StreamThreadNotRunningException` to
> >> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> About check unknown state store names:
> >> >>>>>>>>>>>>>>> Thank you for the hint. I add a new type
> >> >>>>>>>>> `UnknownStateStoreException`
> >> >>>>>>>>>>>> for
> >> >>>>>>>>>>>>>>> this case.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Also, we should still have fatal exception
> >> >>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
> >> >> remove
> >> >>>>> it?
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Thank you point this, already add it again.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> The KIP already updated, please take a look.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> ---
> >> >>>>>>>>>>>>>>> Vito
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >> >
> >>
> >>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, folks,
Just update the KIP, please take a look.
Thanks!
---
Vito
On Fri, Jan 17, 2020 at 9:12 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> I got a lot out of the discussion.
>
> I would like to update KIP-216 base on John's suggestion to remove the
> category.
>
>
> ---
> Vito
>
>
> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> > Nevertheless, if we omit the categorization, it’s moot.
>>
>> Ack.
>>
>> I am fine to remove the middle tier. As John pointed out, it might be
>> weird to have only one concrete exception type per category. We can also
>> explain in detail how to handle each exception in their JavaDocs.
>>
>>
>> -Matthias
>>
>> On 1/16/20 6:38 AM, Bill Bejeck wrote:
>> > Vito,
>> >
>> > Thanks for the updates, the KIP LGTM.
>> >
>> > -Bill
>> >
>> > On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org>
>> wrote:
>> >
>> >> Hi Vito,
>> >>
>> >> Haha, your archive game is on point!
>> >>
>> >> What Matthias said in that email is essentially what I figured was the
>> >> rationale. It makes sense, but the point I was making is that this
>> really
>> >> doesn’t seem like a good way to structure a production app. On the
>> other
>> >> hand, considering the exception fatal has a good chance of avoiding a
>> >> frustrating debug session if you just forgot to call start.
>> >>
>> >> Nevertheless, if we omit the categorization, it’s moot.
>> >>
>> >> It would be easy to add a categorization layer later if we want it, but
>> >> not very easy to change it if we get it wrong.
>> >>
>> >> Thanks for your consideration!
>> >> -John
>> >>
>> >> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
>> >>> Hi John,
>> >>>
>> >>> About `StreamsNotStartedException is strange` --
>> >>> The original idea came from Matthias, two years ago. :)
>> >>> You can reference here:
>> >>>
>> >>
>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>> >>>
>> >>> About omitting the categorization --
>> >>> It looks reasonable. I'm fine with omitting the categorization but not
>> >> very
>> >>> sure it is a good choice.
>> >>> Does any other folks provide opinion?
>> >>>
>> >>>
>> >>> Hi, folks,
>> >>>
>> >>> Just update the KIP-216, please take a look.
>> >>>
>> >>> ---
>> >>> Vito
>> >>>
>> >>>
>> >>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw>
>> wrote:
>> >>>
>> >>>>
>> >>>> Hi, folks,
>> >>>>
>> >>>> Thank you suggestion, really appreciate it. :)
>> >>>> I understand your concern. I'll merge StreamsNotRunningException and
>> >>>> StateStoreNotAvailableException.
>> >>>>
>> >>>>
>> >>>> ---
>> >>>> Vito
>> >>>>
>> >>>>
>> >>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org>
>> >> wrote:
>> >>>>
>> >>>>> Hey Vito,
>> >>>>>
>> >>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>> >>>>> Hopefully, we can close it out soon.
>> >>>>>
>> >>>>> I agree we can drop StreamsNotRunningException in favor of
>> >>>>> just StateStoreNotAvailableException.
>> >>>>>
>> >>>>> Unfortunately, I have some higher-level concerns. The value
>> >>>>> of these exceptions is that they tell you how to handle the
>> >>>>> various situations that can arise while querying a distributed
>> >>>>> data store.
>> >>>>>
>> >>>>> Ideally, as a caller, I should be able to just catch "retriable" or
>> >>>>> "fatal" and handle them appropriately. Otherwise, there's no
>> >>>>> point in having categories, and we should just have all the
>> >>>>> exceptions extend InvalidStateStoreException.
>> >>>>>
>> >>>>> Presently, it's not possible to tell from just the
>> >>>>> "retriable"/"fatal" distinction what to do. You can tell
>> >>>>> from the descriptions of the various exceptions. E.g.:
>> >>>>>
>> >>>>> Retriable:
>> >>>>> * StreamsRebalancingException: the exact same call
>> >>>>> should just be retried until the rebalance is complete
>> >>>>> * StateStoreMigratedException: the store handle is
>> >>>>> now invalid, so you need to re-discover the instance
>> >>>>> and get a new handle on that instance. In other words,
>> >>>>> the query itself may be valid, but the particular method
>> >>>>> invocation on this particular instance has encountered
>> >>>>> a fatal exception.
>> >>>>>
>> >>>>> Fatal:
>> >>>>> * UnknownStateStoreException: this is truly fatal. No amount
>> >>>>> of retrying or re-discovering is going to get you a handle on a
>> >>>>> store that doesn't exist in the cluster.
>> >>>>> * StateStoreNotAvailableException: this is actually recoverable,
>> >>>>> since the store might exist in the cluster, but isn't available
>> on
>> >>>>> this particular instance (which is shut down or whatever).
>> >>>>>
>> >>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
>> >>>>> with omitting the categorization and just having 5 subclasses
>> >>>>> of InvalidStateStoreException. Each of them would tell you
>> >>>>> how to handle them, and it's not too many to really
>> >>>>> understand and handle each one.
>> >>>>>
>> >>>>> If you really want to have a middle tier, I'd recommend:
>> >>>>> * RetryableStateStoreException: the exact same call
>> >>>>> should be repeated.
>> >>>>> * RecoverableStateStoreException: the store handle
>> >>>>> should be discarded and the caller should re-discover
>> >>>>> the location of the store and repeat the query on the
>> >>>>> correct instance.
>> >>>>> * FatalStateStoreException: the query/request is totally
>> >>>>> invalid and will never succeed.
>> >>>>>
>> >>>>> However, attempting to categorize the proposed exceptions
>> >>>>> reveals even problems with this categorization:
>> >>>>> Retriable:
>> >>>>> * StreamsRebalancingException
>> >>>>> Recoverable:
>> >>>>> * StateStoreMigratedException
>> >>>>> * StreamsNotRunningException
>> >>>>> Fatal:
>> >>>>> * UnknownStateStoreException
>> >>>>>
>> >>>>> But StreamsNotStartedException is strange... It means that
>> >>>>> one code path got a handle on a specific KafkaStreams object
>> >>>>> instance and sent it a query before another code path
>> >>>>> invoked the start() method on the exact same object instance.
>> >>>>> It seems like the most likely scenario is that whoever wrote
>> >>>>> the program just forgot to call start() before querying, in
>> >>>>> which case, retrying isn't going to help, and a fatal exception
>> >>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
>> >>>>> experience" problem, and making it fatal would be more
>> >>>>> helpful. Even in a production context, there's no reason not
>> >>>>> to sequence your application startup such that you don't
>> >>>>> accept queries until after Streams is started. Thus, I guess
>> >>>>> I'd categorize it under "fatal".
>> >>>>>
>> >>>>> Regardless of whether you make it fatal or retriable, you'd
>> >>>>> still have a whole category with only one exception in it,
>> >>>>> and the other two categories only have two exceptions.
>> >>>>> Plus, as you pointed out in the KIP, you can't get all
>> >>>>> exceptions in all cases anyway:
>> >>>>> * store() can only throw NotStarted, NotRunning,
>> >>>>> and Unknown
>> >>>>> * actual store queries can only throw Rebalancing,
>> >>>>> Migrated, and NotRunning
>> >>>>>
>> >>>>> Thus, in practice also, there are exactly three categories
>> >>>>> and also exactly three exception types. It doesn't seem
>> >>>>> like there's a great advantage to the categories here. To
>> >>>>> avoid the categorization problem and also to clarify what
>> >>>>> exceptions can actually be thrown in different circumstances,
>> >>>>> it seems like we should just:
>> >>>>> * get rid of the middle tier and make all the exceptions
>> >>>>> extend InvalidStateStoreException
>> >>>>> * drop StateStoreNotAvailableException in favor of
>> >>>>> StreamsNotRunningException
>> >>>>> * clearly document on all public methods which exceptions
>> >>>>> need to be handled
>> >>>>>
>> >>>>> How do you feel about this?
>> >>>>> Thanks,
>> >>>>> -John
>> >>>>>
>> >>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
>> >>>>>> Thanks for KIP Vito.
>> >>>>>>
>> >>>>>> Overall the KIP LGTM, but I'd have to agree with others on merging
>> >> the
>> >>>>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`
>> >>>>> classes.
>> >>>>>>
>> >>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
>> >>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
>> >> when to
>> >>>>>> use the different
>> >>>>>> exceptions. Maybe a good middle ground would be to have a detailed
>> >>>>>> exception message.
>> >>>>>>
>> >>>>>> The KIP freeze is close, so I think if we can agree on this, we can
>> >>>>> wrap up
>> >>>>>> the voting soon.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Bill
>> >>>>>>
>> >>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
>> >> matthias@confluent.io>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Vito,
>> >>>>>>>
>> >>>>>>> It's still unclear to me what the advantage is, to have both
>> >>>>>>> `StreamsNotRunningException` and
>> >> `StateStoreNotAvailableException`?
>> >>>>>>>
>> >>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
>> >> ERROR`
>> >>>>>>> and thus, for a user point of view, why does it matter if the
>> >> store is
>> >>>>>>> closed on not? I don't understand why/how this information would
>> >> be
>> >>>>>>> useful? Do you have a concrete example in mind how a user would
>> >> react
>> >>>>>>> differently to both exceptions?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
>> >> best
>> >>>>> to
>> >>>>>>> actually do this on a per-query basis, ie, have an overload
>> >>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow to
>> >>>>>>> _disable_ the exception and opt-in to query a active store during
>> >>>>>>> recovery. However, as KIP-535 actually introduces this change in
>> >>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
>> >> should be
>> >>>>>>> updated. I'll follow up on the other KIP thread to raise this
>> >> point.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -Matthias
>> >>>>>>>
>> >>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
>> >>>>>>>> Hi, Matthias & Vinoth,
>> >>>>>>>>
>> >>>>>>>> Thanks for the feedback.
>> >>>>>>>>
>> >>>>>>>>> What is still unclear to me is, what we gain by having both
>> >>>>>>>>> `StreamsNotRunningException` and
>> >>>>> `StateStoreNotAvailableException`. Both
>> >>>>>>>>> exception are thrown when KafkaStreams is in state
>> >>>>> PENDING_SHUTDOWN /
>> >>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>> >> if the
>> >>>>>>>>> state store is closed on not -- I can't query it anyway? Maybe
>> >> I
>> >>>>> miss
>> >>>>>>>>> something thought?
>> >>>>>>>>
>> >>>>>>>> Yes, both `StreamsNotRunningException` and
>> >>>>>>>> `StateStoreNotAvailableException` are fatal exception.
>> >>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
>> >> state
>> >>>>>>> store
>> >>>>>>>> related.
>> >>>>>>>> I think it would be helpful that if user need to distinguish
>> >> these
>> >>>>> two
>> >>>>>>>> different case to handle it.
>> >>>>>>>>
>> >>>>>>>> I'm not very sure, does that make sense?
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ---
>> >>>>>>>> Vito
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
>> >> vinoth@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> +1 on merging `StreamsNotRunningException` and
>> >>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
>> >>>>> anyway. IMO
>> >>>>>>>>> its best to have these exceptions be about the state store
>> >> (and not
>> >>>>>>> streams
>> >>>>>>>>> state), to easier understanding.
>> >>>>>>>>>
>> >>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
>> >>>>> rebalancing
>> >>>>>>>>> state. So do we need the StreamsRebalancingException?
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
>> >> matthias@confluent.io>
>> >>>>>>> wrote:
>> >>>>>>>>>> Sorry that I dropped the ball on this...
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
>> >> start
>> >>>>> a
>> >>>>>>> VOTE
>> >>>>>>>>>> thread.
>> >>>>>>>>>>
>> >>>>>>>>>> What is still unclear to me is, what we gain by having both
>> >>>>>>>>>> `StreamsNotRunningException` and
>> >>>>> `StateStoreNotAvailableException`.
>> >>>>>>> Both
>> >>>>>>>>>> exception are thrown when KafkaStreams is in state
>> >>>>> PENDING_SHUTDOWN /
>> >>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>> >> if
>> >>>>> the
>> >>>>>>>>>> state store is closed on not -- I can't query it anyway?
>> >> Maybe I
>> >>>>> miss
>> >>>>>>>>>> something thought?
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> -Matthias
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>> >>>>>>>>>>> Sorry for the late reply, thanks for the review.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>> About `StateStoreMigratedException`:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
>> >>>>> might be
>> >>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
>> >>>>> back to
>> >>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>> >> store
>> >>>>>>> handle.
>> >>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>> >>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>> >>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>> >>>>> `RUNNING`
>> >>>>>>>>> state.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thank you point this, already updated.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Why do we need to distinguish between
>> >>>>>>> `KafkaStreamsNotRunningException`
>> >>>>>>>>>>>> and `StateStoreNotAvailableException`?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
>> >>>>> reasons, I
>> >>>>>>>>> think
>> >>>>>>>>>>> it would be helpful that the
>> >>>>>>>>>>> user can distinguish whether it is caused by the state store
>> >>>>> closed.
>> >>>>>>>>>>> (Maybe I am wrong...)
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
>> >> and
>> >>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
>> >> the
>> >>>>>>>>> instance,
>> >>>>>>>>>>>> because that is the level of granularity in which we
>> >>>>> enable/disable
>> >>>>>>>>> IQ atm.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Totally agree. Do you mean the naming of state store
>> >> exceptions?
>> >>>>>>>>>>> I don't have special reason to distinguish these two.
>> >>>>>>>>>>> Your suggestion look more reasonable for the exception
>> >> naming.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
>> >> user
>> >>>>> need
>> >>>>>>>>> to
>> >>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
>> >>>>> handle is
>> >>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
>> >>>>>>> difference
>> >>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>> >> "blind"
>> >>>>> retries
>> >>>>>>>>>>>> that either resolve (if the store is still on the same
>> >> instance
>> >>>>> after
>> >>>>>>>>>>>> rebalancing finishes, or changes to
>> >>>>> `StateStoreMigratedException` if
>> >>>>>>>>> the
>> >>>>>>>>>>>> store was migrated away during rebalancing).
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Nice, it's great! Thank you.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> The KIP already updated, please take a look. :)
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
>> >>>>>>> matthias@confluent.io
>> >>>>>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Any update on this KIP?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>> >>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
>> >> busy.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> About `StateStoreMigratedException`:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
>> >>>>> might be
>> >>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
>> >>>>> back to
>> >>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>> >> store
>> >>>>>>>>> handle.
>> >>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>> >>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>> >>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>> >>>>> `RUNNING`
>> >>>>>>>>>>>> state.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Why do we need to distinguish between
>> >>>>>>>>> `KafkaStreamsNotRunningException`
>> >>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
>> >> instance and
>> >>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
>> >> the
>> >>>>>>>>> instance,
>> >>>>>>>>>>>>> because that is the level of granularity in which we
>> >>>>> enable/disable
>> >>>>>>>>> IQ
>> >>>>>>>>>>>> atm.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
>> >>>>> user
>> >>>>>>>>> need to
>> >>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
>> >>>>> handle is
>> >>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
>> >>>>>>>>> difference
>> >>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>> >> "blind"
>> >>>>>>> retries
>> >>>>>>>>>>>>> that either resolve (if the store is still on the same
>> >> instance
>> >>>>>>> after
>> >>>>>>>>>>>>> rebalancing finishes, or changes to
>> >>>>> `StateStoreMigratedException` if
>> >>>>>>>>> the
>> >>>>>>>>>>>>> store was migrated away during rebalancing).
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>> >>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9>
>> >> <https://shorturl.at/CDNT9>
>> >>>>> <https://shorturl.at/CDNT9>
>> >>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>>>>> <https://shorturl.at/CDNT9>
>> >>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Please use the following instead:
>> >> https://shorturl.at/bkKQU
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
>> >>>>> vito@is-land.com.tw>
>> >>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks, Matthias!
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
>> >>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>> >>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
>> >>>>> handed out
>> >>>>>>>>> the
>> >>>>>>>>>>>>>>> store handle. We may throw
>> >> `KafkaStreamsNotRunningException`
>> >>>>> or
>> >>>>>>>>>>>>>>> `StateStoreMigratedException`.
>> >>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
>> >> not
>> >>>>>>>>> running(
>> >>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
>> >>>>> `StateStoreMigratedException`
>> >>>>>>>>> when
>> >>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
>> >> we
>> >>>>> do not
>> >>>>>>>>> need
>> >>>>>>>>>>>> to
>> >>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
>> >>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I understand your point. I rename
>> >>>>>>>>> `StreamThreadNotRunningException` to
>> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> About check unknown state store names:
>> >>>>>>>>>>>>>>> Thank you for the hint. I add a new type
>> >>>>>>>>> `UnknownStateStoreException`
>> >>>>>>>>>>>> for
>> >>>>>>>>>>>>>>> this case.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Also, we should still have fatal exception
>> >>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
>> >> remove
>> >>>>> it?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thank you point this, already add it again.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> The KIP already updated, please take a look.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors
Posted by John Roesler <vv...@apache.org>.
Thanks for your patience, Vito!
-John
On Thu, Jan 16, 2020, at 19:12, Vito Jeng wrote:
> Thanks Bill, John and Matthias. Glad you guys joined this discussion.
> I got a lot out of the discussion.
>
> I would like to update KIP-216 base on John's suggestion to remove the
> category.
>
>
> ---
> Vito
>
>
> On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > > Nevertheless, if we omit the categorization, it’s moot.
> >
> > Ack.
> >
> > I am fine to remove the middle tier. As John pointed out, it might be
> > weird to have only one concrete exception type per category. We can also
> > explain in detail how to handle each exception in their JavaDocs.
> >
> >
> > -Matthias
> >
> > On 1/16/20 6:38 AM, Bill Bejeck wrote:
> > > Vito,
> > >
> > > Thanks for the updates, the KIP LGTM.
> > >
> > > -Bill
> > >
> > > On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org>
> > wrote:
> > >
> > >> Hi Vito,
> > >>
> > >> Haha, your archive game is on point!
> > >>
> > >> What Matthias said in that email is essentially what I figured was the
> > >> rationale. It makes sense, but the point I was making is that this
> > really
> > >> doesn’t seem like a good way to structure a production app. On the other
> > >> hand, considering the exception fatal has a good chance of avoiding a
> > >> frustrating debug session if you just forgot to call start.
> > >>
> > >> Nevertheless, if we omit the categorization, it’s moot.
> > >>
> > >> It would be easy to add a categorization layer later if we want it, but
> > >> not very easy to change it if we get it wrong.
> > >>
> > >> Thanks for your consideration!
> > >> -John
> > >>
> > >> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> > >>> Hi John,
> > >>>
> > >>> About `StreamsNotStartedException is strange` --
> > >>> The original idea came from Matthias, two years ago. :)
> > >>> You can reference here:
> > >>>
> > >>
> > https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> > >>>
> > >>> About omitting the categorization --
> > >>> It looks reasonable. I'm fine with omitting the categorization but not
> > >> very
> > >>> sure it is a good choice.
> > >>> Does any other folks provide opinion?
> > >>>
> > >>>
> > >>> Hi, folks,
> > >>>
> > >>> Just update the KIP-216, please take a look.
> > >>>
> > >>> ---
> > >>> Vito
> > >>>
> > >>>
> > >>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> > >>>
> > >>>>
> > >>>> Hi, folks,
> > >>>>
> > >>>> Thank you suggestion, really appreciate it. :)
> > >>>> I understand your concern. I'll merge StreamsNotRunningException and
> > >>>> StateStoreNotAvailableException.
> > >>>>
> > >>>>
> > >>>> ---
> > >>>> Vito
> > >>>>
> > >>>>
> > >>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org>
> > >> wrote:
> > >>>>
> > >>>>> Hey Vito,
> > >>>>>
> > >>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
> > >>>>> Hopefully, we can close it out soon.
> > >>>>>
> > >>>>> I agree we can drop StreamsNotRunningException in favor of
> > >>>>> just StateStoreNotAvailableException.
> > >>>>>
> > >>>>> Unfortunately, I have some higher-level concerns. The value
> > >>>>> of these exceptions is that they tell you how to handle the
> > >>>>> various situations that can arise while querying a distributed
> > >>>>> data store.
> > >>>>>
> > >>>>> Ideally, as a caller, I should be able to just catch "retriable" or
> > >>>>> "fatal" and handle them appropriately. Otherwise, there's no
> > >>>>> point in having categories, and we should just have all the
> > >>>>> exceptions extend InvalidStateStoreException.
> > >>>>>
> > >>>>> Presently, it's not possible to tell from just the
> > >>>>> "retriable"/"fatal" distinction what to do. You can tell
> > >>>>> from the descriptions of the various exceptions. E.g.:
> > >>>>>
> > >>>>> Retriable:
> > >>>>> * StreamsRebalancingException: the exact same call
> > >>>>> should just be retried until the rebalance is complete
> > >>>>> * StateStoreMigratedException: the store handle is
> > >>>>> now invalid, so you need to re-discover the instance
> > >>>>> and get a new handle on that instance. In other words,
> > >>>>> the query itself may be valid, but the particular method
> > >>>>> invocation on this particular instance has encountered
> > >>>>> a fatal exception.
> > >>>>>
> > >>>>> Fatal:
> > >>>>> * UnknownStateStoreException: this is truly fatal. No amount
> > >>>>> of retrying or re-discovering is going to get you a handle on a
> > >>>>> store that doesn't exist in the cluster.
> > >>>>> * StateStoreNotAvailableException: this is actually recoverable,
> > >>>>> since the store might exist in the cluster, but isn't available
> > on
> > >>>>> this particular instance (which is shut down or whatever).
> > >>>>>
> > >>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> > >>>>> with omitting the categorization and just having 5 subclasses
> > >>>>> of InvalidStateStoreException. Each of them would tell you
> > >>>>> how to handle them, and it's not too many to really
> > >>>>> understand and handle each one.
> > >>>>>
> > >>>>> If you really want to have a middle tier, I'd recommend:
> > >>>>> * RetryableStateStoreException: the exact same call
> > >>>>> should be repeated.
> > >>>>> * RecoverableStateStoreException: the store handle
> > >>>>> should be discarded and the caller should re-discover
> > >>>>> the location of the store and repeat the query on the
> > >>>>> correct instance.
> > >>>>> * FatalStateStoreException: the query/request is totally
> > >>>>> invalid and will never succeed.
> > >>>>>
> > >>>>> However, attempting to categorize the proposed exceptions
> > >>>>> reveals even problems with this categorization:
> > >>>>> Retriable:
> > >>>>> * StreamsRebalancingException
> > >>>>> Recoverable:
> > >>>>> * StateStoreMigratedException
> > >>>>> * StreamsNotRunningException
> > >>>>> Fatal:
> > >>>>> * UnknownStateStoreException
> > >>>>>
> > >>>>> But StreamsNotStartedException is strange... It means that
> > >>>>> one code path got a handle on a specific KafkaStreams object
> > >>>>> instance and sent it a query before another code path
> > >>>>> invoked the start() method on the exact same object instance.
> > >>>>> It seems like the most likely scenario is that whoever wrote
> > >>>>> the program just forgot to call start() before querying, in
> > >>>>> which case, retrying isn't going to help, and a fatal exception
> > >>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
> > >>>>> experience" problem, and making it fatal would be more
> > >>>>> helpful. Even in a production context, there's no reason not
> > >>>>> to sequence your application startup such that you don't
> > >>>>> accept queries until after Streams is started. Thus, I guess
> > >>>>> I'd categorize it under "fatal".
> > >>>>>
> > >>>>> Regardless of whether you make it fatal or retriable, you'd
> > >>>>> still have a whole category with only one exception in it,
> > >>>>> and the other two categories only have two exceptions.
> > >>>>> Plus, as you pointed out in the KIP, you can't get all
> > >>>>> exceptions in all cases anyway:
> > >>>>> * store() can only throw NotStarted, NotRunning,
> > >>>>> and Unknown
> > >>>>> * actual store queries can only throw Rebalancing,
> > >>>>> Migrated, and NotRunning
> > >>>>>
> > >>>>> Thus, in practice also, there are exactly three categories
> > >>>>> and also exactly three exception types. It doesn't seem
> > >>>>> like there's a great advantage to the categories here. To
> > >>>>> avoid the categorization problem and also to clarify what
> > >>>>> exceptions can actually be thrown in different circumstances,
> > >>>>> it seems like we should just:
> > >>>>> * get rid of the middle tier and make all the exceptions
> > >>>>> extend InvalidStateStoreException
> > >>>>> * drop StateStoreNotAvailableException in favor of
> > >>>>> StreamsNotRunningException
> > >>>>> * clearly document on all public methods which exceptions
> > >>>>> need to be handled
> > >>>>>
> > >>>>> How do you feel about this?
> > >>>>> Thanks,
> > >>>>> -John
> > >>>>>
> > >>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> > >>>>>> Thanks for KIP Vito.
> > >>>>>>
> > >>>>>> Overall the KIP LGTM, but I'd have to agree with others on merging
> > >> the
> > >>>>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`
> > >>>>> classes.
> > >>>>>>
> > >>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> > >>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
> > >> when to
> > >>>>>> use the different
> > >>>>>> exceptions. Maybe a good middle ground would be to have a detailed
> > >>>>>> exception message.
> > >>>>>>
> > >>>>>> The KIP freeze is close, so I think if we can agree on this, we can
> > >>>>> wrap up
> > >>>>>> the voting soon.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Bill
> > >>>>>>
> > >>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
> > >> matthias@confluent.io>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Vito,
> > >>>>>>>
> > >>>>>>> It's still unclear to me what the advantage is, to have both
> > >>>>>>> `StreamsNotRunningException` and
> > >> `StateStoreNotAvailableException`?
> > >>>>>>>
> > >>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
> > >> ERROR`
> > >>>>>>> and thus, for a user point of view, why does it matter if the
> > >> store is
> > >>>>>>> closed on not? I don't understand why/how this information would
> > >> be
> > >>>>>>> useful? Do you have a concrete example in mind how a user would
> > >> react
> > >>>>>>> differently to both exceptions?
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
> > >> best
> > >>>>> to
> > >>>>>>> actually do this on a per-query basis, ie, have an overload
> > >>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow to
> > >>>>>>> _disable_ the exception and opt-in to query a active store during
> > >>>>>>> recovery. However, as KIP-535 actually introduces this change in
> > >>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
> > >> should be
> > >>>>>>> updated. I'll follow up on the other KIP thread to raise this
> > >> point.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> -Matthias
> > >>>>>>>
> > >>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
> > >>>>>>>> Hi, Matthias & Vinoth,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback.
> > >>>>>>>>
> > >>>>>>>>> What is still unclear to me is, what we gain by having both
> > >>>>>>>>> `StreamsNotRunningException` and
> > >>>>> `StateStoreNotAvailableException`. Both
> > >>>>>>>>> exception are thrown when KafkaStreams is in state
> > >>>>> PENDING_SHUTDOWN /
> > >>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> > >> if the
> > >>>>>>>>> state store is closed on not -- I can't query it anyway? Maybe
> > >> I
> > >>>>> miss
> > >>>>>>>>> something thought?
> > >>>>>>>>
> > >>>>>>>> Yes, both `StreamsNotRunningException` and
> > >>>>>>>> `StateStoreNotAvailableException` are fatal exception.
> > >>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
> > >> state
> > >>>>>>> store
> > >>>>>>>> related.
> > >>>>>>>> I think it would be helpful that if user need to distinguish
> > >> these
> > >>>>> two
> > >>>>>>>> different case to handle it.
> > >>>>>>>>
> > >>>>>>>> I'm not very sure, does that make sense?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> ---
> > >>>>>>>> Vito
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
> > >> vinoth@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> +1 on merging `StreamsNotRunningException` and
> > >>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
> > >>>>> anyway. IMO
> > >>>>>>>>> its best to have these exceptions be about the state store
> > >> (and not
> > >>>>>>> streams
> > >>>>>>>>> state), to easier understanding.
> > >>>>>>>>>
> > >>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
> > >>>>> rebalancing
> > >>>>>>>>> state. So do we need the StreamsRebalancingException?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
> > >> matthias@confluent.io>
> > >>>>>>> wrote:
> > >>>>>>>>>> Sorry that I dropped the ball on this...
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
> > >> start
> > >>>>> a
> > >>>>>>> VOTE
> > >>>>>>>>>> thread.
> > >>>>>>>>>>
> > >>>>>>>>>> What is still unclear to me is, what we gain by having both
> > >>>>>>>>>> `StreamsNotRunningException` and
> > >>>>> `StateStoreNotAvailableException`.
> > >>>>>>> Both
> > >>>>>>>>>> exception are thrown when KafkaStreams is in state
> > >>>>> PENDING_SHUTDOWN /
> > >>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> > >> if
> > >>>>> the
> > >>>>>>>>>> state store is closed on not -- I can't query it anyway?
> > >> Maybe I
> > >>>>> miss
> > >>>>>>>>>> something thought?
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> -Matthias
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> > >>>>>>>>>>> Sorry for the late reply, thanks for the review.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> About `StateStoreMigratedException`:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
> > >>>>> might be
> > >>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
> > >>>>> back to
> > >>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> > >> store
> > >>>>>>> handle.
> > >>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> > >>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> > >>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> > >>>>> `RUNNING`
> > >>>>>>>>> state.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thank you point this, already updated.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Why do we need to distinguish between
> > >>>>>>> `KafkaStreamsNotRunningException`
> > >>>>>>>>>>>> and `StateStoreNotAvailableException`?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
> > >>>>> reasons, I
> > >>>>>>>>> think
> > >>>>>>>>>>> it would be helpful that the
> > >>>>>>>>>>> user can distinguish whether it is caused by the state store
> > >>>>> closed.
> > >>>>>>>>>>> (Maybe I am wrong...)
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
> > >> and
> > >>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
> > >> the
> > >>>>>>>>> instance,
> > >>>>>>>>>>>> because that is the level of granularity in which we
> > >>>>> enable/disable
> > >>>>>>>>> IQ atm.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Totally agree. Do you mean the naming of state store
> > >> exceptions?
> > >>>>>>>>>>> I don't have special reason to distinguish these two.
> > >>>>>>>>>>> Your suggestion look more reasonable for the exception
> > >> naming.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
> > >> user
> > >>>>> need
> > >>>>>>>>> to
> > >>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
> > >>>>> handle is
> > >>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
> > >>>>>>> difference
> > >>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> > >> "blind"
> > >>>>> retries
> > >>>>>>>>>>>> that either resolve (if the store is still on the same
> > >> instance
> > >>>>> after
> > >>>>>>>>>>>> rebalancing finishes, or changes to
> > >>>>> `StateStoreMigratedException` if
> > >>>>>>>>> the
> > >>>>>>>>>>>> store was migrated away during rebalancing).
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Nice, it's great! Thank you.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> The KIP already updated, please take a look. :)
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> > >>>>>>> matthias@confluent.io
> > >>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Any update on this KIP?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > >>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
> > >> busy.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> About `StateStoreMigratedException`:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
> > >>>>> might be
> > >>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
> > >>>>> back to
> > >>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> > >> store
> > >>>>>>>>> handle.
> > >>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> > >>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> > >>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> > >>>>> `RUNNING`
> > >>>>>>>>>>>> state.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Why do we need to distinguish between
> > >>>>>>>>> `KafkaStreamsNotRunningException`
> > >>>>>>>>>>>>> and `StateStoreNotAvailableException`?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
> > >> instance and
> > >>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
> > >> the
> > >>>>>>>>> instance,
> > >>>>>>>>>>>>> because that is the level of granularity in which we
> > >>>>> enable/disable
> > >>>>>>>>> IQ
> > >>>>>>>>>>>> atm.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
> > >>>>> user
> > >>>>>>>>> need to
> > >>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
> > >>>>> handle is
> > >>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
> > >>>>>>>>> difference
> > >>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> > >> "blind"
> > >>>>>>> retries
> > >>>>>>>>>>>>> that either resolve (if the store is still on the same
> > >> instance
> > >>>>>>> after
> > >>>>>>>>>>>>> rebalancing finishes, or changes to
> > >>>>> `StateStoreMigratedException` if
> > >>>>>>>>> the
> > >>>>>>>>>>>>> store was migrated away during rebalancing).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> -Matthias
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> > >>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> > <https://shorturl.at/CDNT9>
> > >> <https://shorturl.at/CDNT9>
> > >>>>> <https://shorturl.at/CDNT9>
> > >>>>>>> <https://shorturl.at/CDNT9>
> > >>>>>>>>> <https://shorturl.at/CDNT9>
> > >>>>>>>>>>>> <https://shorturl.at/CDNT9>
> > >>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Please use the following instead:
> > >> https://shorturl.at/bkKQU
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>> Vito
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
> > >>>>> vito@is-land.com.tw>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thanks, Matthias!
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
> > >>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> > >>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
> > >>>>> handed out
> > >>>>>>>>> the
> > >>>>>>>>>>>>>>> store handle. We may throw
> > >> `KafkaStreamsNotRunningException`
> > >>>>> or
> > >>>>>>>>>>>>>>> `StateStoreMigratedException`.
> > >>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> > >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
> > >> not
> > >>>>>>>>> running(
> > >>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
> > >>>>> `StateStoreMigratedException`
> > >>>>>>>>> when
> > >>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
> > >> we
> > >>>>> do not
> > >>>>>>>>> need
> > >>>>>>>>>>>> to
> > >>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
> > >>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> I understand your point. I rename
> > >>>>>>>>> `StreamThreadNotRunningException` to
> > >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> About check unknown state store names:
> > >>>>>>>>>>>>>>> Thank you for the hint. I add a new type
> > >>>>>>>>> `UnknownStateStoreException`
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> this case.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Also, we should still have fatal exception
> > >>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
> > >> remove
> > >>>>> it?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Thank you point this, already add it again.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> The KIP already updated, please take a look.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>> Vito
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Thanks Bill, John and Matthias. Glad you guys joined this discussion.
I got a lot out of the discussion.
I would like to update KIP-216 base on John's suggestion to remove the
category.
---
Vito
On Fri, Jan 17, 2020 at 2:30 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> > Nevertheless, if we omit the categorization, it’s moot.
>
> Ack.
>
> I am fine to remove the middle tier. As John pointed out, it might be
> weird to have only one concrete exception type per category. We can also
> explain in detail how to handle each exception in their JavaDocs.
>
>
> -Matthias
>
> On 1/16/20 6:38 AM, Bill Bejeck wrote:
> > Vito,
> >
> > Thanks for the updates, the KIP LGTM.
> >
> > -Bill
> >
> > On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org>
> wrote:
> >
> >> Hi Vito,
> >>
> >> Haha, your archive game is on point!
> >>
> >> What Matthias said in that email is essentially what I figured was the
> >> rationale. It makes sense, but the point I was making is that this
> really
> >> doesn’t seem like a good way to structure a production app. On the other
> >> hand, considering the exception fatal has a good chance of avoiding a
> >> frustrating debug session if you just forgot to call start.
> >>
> >> Nevertheless, if we omit the categorization, it’s moot.
> >>
> >> It would be easy to add a categorization layer later if we want it, but
> >> not very easy to change it if we get it wrong.
> >>
> >> Thanks for your consideration!
> >> -John
> >>
> >> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> >>> Hi John,
> >>>
> >>> About `StreamsNotStartedException is strange` --
> >>> The original idea came from Matthias, two years ago. :)
> >>> You can reference here:
> >>>
> >>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> >>>
> >>> About omitting the categorization --
> >>> It looks reasonable. I'm fine with omitting the categorization but not
> >> very
> >>> sure it is a good choice.
> >>> Does any other folks provide opinion?
> >>>
> >>>
> >>> Hi, folks,
> >>>
> >>> Just update the KIP-216, please take a look.
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> >>>
> >>>>
> >>>> Hi, folks,
> >>>>
> >>>> Thank you suggestion, really appreciate it. :)
> >>>> I understand your concern. I'll merge StreamsNotRunningException and
> >>>> StateStoreNotAvailableException.
> >>>>
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>>
> >>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org>
> >> wrote:
> >>>>
> >>>>> Hey Vito,
> >>>>>
> >>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
> >>>>> Hopefully, we can close it out soon.
> >>>>>
> >>>>> I agree we can drop StreamsNotRunningException in favor of
> >>>>> just StateStoreNotAvailableException.
> >>>>>
> >>>>> Unfortunately, I have some higher-level concerns. The value
> >>>>> of these exceptions is that they tell you how to handle the
> >>>>> various situations that can arise while querying a distributed
> >>>>> data store.
> >>>>>
> >>>>> Ideally, as a caller, I should be able to just catch "retriable" or
> >>>>> "fatal" and handle them appropriately. Otherwise, there's no
> >>>>> point in having categories, and we should just have all the
> >>>>> exceptions extend InvalidStateStoreException.
> >>>>>
> >>>>> Presently, it's not possible to tell from just the
> >>>>> "retriable"/"fatal" distinction what to do. You can tell
> >>>>> from the descriptions of the various exceptions. E.g.:
> >>>>>
> >>>>> Retriable:
> >>>>> * StreamsRebalancingException: the exact same call
> >>>>> should just be retried until the rebalance is complete
> >>>>> * StateStoreMigratedException: the store handle is
> >>>>> now invalid, so you need to re-discover the instance
> >>>>> and get a new handle on that instance. In other words,
> >>>>> the query itself may be valid, but the particular method
> >>>>> invocation on this particular instance has encountered
> >>>>> a fatal exception.
> >>>>>
> >>>>> Fatal:
> >>>>> * UnknownStateStoreException: this is truly fatal. No amount
> >>>>> of retrying or re-discovering is going to get you a handle on a
> >>>>> store that doesn't exist in the cluster.
> >>>>> * StateStoreNotAvailableException: this is actually recoverable,
> >>>>> since the store might exist in the cluster, but isn't available
> on
> >>>>> this particular instance (which is shut down or whatever).
> >>>>>
> >>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> >>>>> with omitting the categorization and just having 5 subclasses
> >>>>> of InvalidStateStoreException. Each of them would tell you
> >>>>> how to handle them, and it's not too many to really
> >>>>> understand and handle each one.
> >>>>>
> >>>>> If you really want to have a middle tier, I'd recommend:
> >>>>> * RetryableStateStoreException: the exact same call
> >>>>> should be repeated.
> >>>>> * RecoverableStateStoreException: the store handle
> >>>>> should be discarded and the caller should re-discover
> >>>>> the location of the store and repeat the query on the
> >>>>> correct instance.
> >>>>> * FatalStateStoreException: the query/request is totally
> >>>>> invalid and will never succeed.
> >>>>>
> >>>>> However, attempting to categorize the proposed exceptions
> >>>>> reveals even problems with this categorization:
> >>>>> Retriable:
> >>>>> * StreamsRebalancingException
> >>>>> Recoverable:
> >>>>> * StateStoreMigratedException
> >>>>> * StreamsNotRunningException
> >>>>> Fatal:
> >>>>> * UnknownStateStoreException
> >>>>>
> >>>>> But StreamsNotStartedException is strange... It means that
> >>>>> one code path got a handle on a specific KafkaStreams object
> >>>>> instance and sent it a query before another code path
> >>>>> invoked the start() method on the exact same object instance.
> >>>>> It seems like the most likely scenario is that whoever wrote
> >>>>> the program just forgot to call start() before querying, in
> >>>>> which case, retrying isn't going to help, and a fatal exception
> >>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
> >>>>> experience" problem, and making it fatal would be more
> >>>>> helpful. Even in a production context, there's no reason not
> >>>>> to sequence your application startup such that you don't
> >>>>> accept queries until after Streams is started. Thus, I guess
> >>>>> I'd categorize it under "fatal".
> >>>>>
> >>>>> Regardless of whether you make it fatal or retriable, you'd
> >>>>> still have a whole category with only one exception in it,
> >>>>> and the other two categories only have two exceptions.
> >>>>> Plus, as you pointed out in the KIP, you can't get all
> >>>>> exceptions in all cases anyway:
> >>>>> * store() can only throw NotStarted, NotRunning,
> >>>>> and Unknown
> >>>>> * actual store queries can only throw Rebalancing,
> >>>>> Migrated, and NotRunning
> >>>>>
> >>>>> Thus, in practice also, there are exactly three categories
> >>>>> and also exactly three exception types. It doesn't seem
> >>>>> like there's a great advantage to the categories here. To
> >>>>> avoid the categorization problem and also to clarify what
> >>>>> exceptions can actually be thrown in different circumstances,
> >>>>> it seems like we should just:
> >>>>> * get rid of the middle tier and make all the exceptions
> >>>>> extend InvalidStateStoreException
> >>>>> * drop StateStoreNotAvailableException in favor of
> >>>>> StreamsNotRunningException
> >>>>> * clearly document on all public methods which exceptions
> >>>>> need to be handled
> >>>>>
> >>>>> How do you feel about this?
> >>>>> Thanks,
> >>>>> -John
> >>>>>
> >>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> >>>>>> Thanks for KIP Vito.
> >>>>>>
> >>>>>> Overall the KIP LGTM, but I'd have to agree with others on merging
> >> the
> >>>>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`
> >>>>> classes.
> >>>>>>
> >>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> >>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
> >> when to
> >>>>>> use the different
> >>>>>> exceptions. Maybe a good middle ground would be to have a detailed
> >>>>>> exception message.
> >>>>>>
> >>>>>> The KIP freeze is close, so I think if we can agree on this, we can
> >>>>> wrap up
> >>>>>> the voting soon.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Bill
> >>>>>>
> >>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
> >> matthias@confluent.io>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Vito,
> >>>>>>>
> >>>>>>> It's still unclear to me what the advantage is, to have both
> >>>>>>> `StreamsNotRunningException` and
> >> `StateStoreNotAvailableException`?
> >>>>>>>
> >>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
> >> ERROR`
> >>>>>>> and thus, for a user point of view, why does it matter if the
> >> store is
> >>>>>>> closed on not? I don't understand why/how this information would
> >> be
> >>>>>>> useful? Do you have a concrete example in mind how a user would
> >> react
> >>>>>>> differently to both exceptions?
> >>>>>>>
> >>>>>>>
> >>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
> >> best
> >>>>> to
> >>>>>>> actually do this on a per-query basis, ie, have an overload
> >>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow to
> >>>>>>> _disable_ the exception and opt-in to query a active store during
> >>>>>>> recovery. However, as KIP-535 actually introduces this change in
> >>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
> >> should be
> >>>>>>> updated. I'll follow up on the other KIP thread to raise this
> >> point.
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
> >>>>>>>> Hi, Matthias & Vinoth,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback.
> >>>>>>>>
> >>>>>>>>> What is still unclear to me is, what we gain by having both
> >>>>>>>>> `StreamsNotRunningException` and
> >>>>> `StateStoreNotAvailableException`. Both
> >>>>>>>>> exception are thrown when KafkaStreams is in state
> >>>>> PENDING_SHUTDOWN /
> >>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> >> if the
> >>>>>>>>> state store is closed on not -- I can't query it anyway? Maybe
> >> I
> >>>>> miss
> >>>>>>>>> something thought?
> >>>>>>>>
> >>>>>>>> Yes, both `StreamsNotRunningException` and
> >>>>>>>> `StateStoreNotAvailableException` are fatal exception.
> >>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
> >> state
> >>>>>>> store
> >>>>>>>> related.
> >>>>>>>> I think it would be helpful that if user need to distinguish
> >> these
> >>>>> two
> >>>>>>>> different case to handle it.
> >>>>>>>>
> >>>>>>>> I'm not very sure, does that make sense?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Vito
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
> >> vinoth@apache.org>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> +1 on merging `StreamsNotRunningException` and
> >>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
> >>>>> anyway. IMO
> >>>>>>>>> its best to have these exceptions be about the state store
> >> (and not
> >>>>>>> streams
> >>>>>>>>> state), to easier understanding.
> >>>>>>>>>
> >>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
> >>>>> rebalancing
> >>>>>>>>> state. So do we need the StreamsRebalancingException?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
> >> matthias@confluent.io>
> >>>>>>> wrote:
> >>>>>>>>>> Sorry that I dropped the ball on this...
> >>>>>>>>>>
> >>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
> >> start
> >>>>> a
> >>>>>>> VOTE
> >>>>>>>>>> thread.
> >>>>>>>>>>
> >>>>>>>>>> What is still unclear to me is, what we gain by having both
> >>>>>>>>>> `StreamsNotRunningException` and
> >>>>> `StateStoreNotAvailableException`.
> >>>>>>> Both
> >>>>>>>>>> exception are thrown when KafkaStreams is in state
> >>>>> PENDING_SHUTDOWN /
> >>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> >> if
> >>>>> the
> >>>>>>>>>> state store is closed on not -- I can't query it anyway?
> >> Maybe I
> >>>>> miss
> >>>>>>>>>> something thought?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> >>>>>>>>>>> Sorry for the late reply, thanks for the review.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> About `StateStoreMigratedException`:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
> >>>>> might be
> >>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
> >>>>> back to
> >>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> >> store
> >>>>>>> handle.
> >>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> >>>>> `RUNNING`
> >>>>>>>>> state.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thank you point this, already updated.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Why do we need to distinguish between
> >>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
> >>>>> reasons, I
> >>>>>>>>> think
> >>>>>>>>>>> it would be helpful that the
> >>>>>>>>>>> user can distinguish whether it is caused by the state store
> >>>>> closed.
> >>>>>>>>>>> (Maybe I am wrong...)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
> >> and
> >>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
> >> the
> >>>>>>>>> instance,
> >>>>>>>>>>>> because that is the level of granularity in which we
> >>>>> enable/disable
> >>>>>>>>> IQ atm.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Totally agree. Do you mean the naming of state store
> >> exceptions?
> >>>>>>>>>>> I don't have special reason to distinguish these two.
> >>>>>>>>>>> Your suggestion look more reasonable for the exception
> >> naming.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
> >> user
> >>>>> need
> >>>>>>>>> to
> >>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
> >>>>> handle is
> >>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
> >>>>>>> difference
> >>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >> "blind"
> >>>>> retries
> >>>>>>>>>>>> that either resolve (if the store is still on the same
> >> instance
> >>>>> after
> >>>>>>>>>>>> rebalancing finishes, or changes to
> >>>>> `StateStoreMigratedException` if
> >>>>>>>>> the
> >>>>>>>>>>>> store was migrated away during rebalancing).
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Nice, it's great! Thank you.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> The KIP already updated, please take a look. :)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> >>>>>>> matthias@confluent.io
> >>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Any update on this KIP?
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
> >> busy.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> About `StateStoreMigratedException`:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
> >>>>> might be
> >>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
> >>>>> back to
> >>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
> >> store
> >>>>>>>>> handle.
> >>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
> >>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
> >>>>> `RUNNING`
> >>>>>>>>>>>> state.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Why do we need to distinguish between
> >>>>>>>>> `KafkaStreamsNotRunningException`
> >>>>>>>>>>>>> and `StateStoreNotAvailableException`?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
> >> instance and
> >>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
> >> the
> >>>>>>>>> instance,
> >>>>>>>>>>>>> because that is the level of granularity in which we
> >>>>> enable/disable
> >>>>>>>>> IQ
> >>>>>>>>>>>> atm.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
> >>>>> user
> >>>>>>>>> need to
> >>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
> >>>>> handle is
> >>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
> >>>>>>>>> difference
> >>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
> >> "blind"
> >>>>>>> retries
> >>>>>>>>>>>>> that either resolve (if the store is still on the same
> >> instance
> >>>>>>> after
> >>>>>>>>>>>>> rebalancing finishes, or changes to
> >>>>> `StateStoreMigratedException` if
> >>>>>>>>> the
> >>>>>>>>>>>>> store was migrated away during rebalancing).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >> <https://shorturl.at/CDNT9>
> >>>>> <https://shorturl.at/CDNT9>
> >>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>> <https://shorturl.at/CDNT9>
> >>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Please use the following instead:
> >> https://shorturl.at/bkKQU
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
> >>>>> vito@is-land.com.tw>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, Matthias!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
> >>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
> >>>>> handed out
> >>>>>>>>> the
> >>>>>>>>>>>>>>> store handle. We may throw
> >> `KafkaStreamsNotRunningException`
> >>>>> or
> >>>>>>>>>>>>>>> `StateStoreMigratedException`.
> >>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
> >> not
> >>>>>>>>> running(
> >>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
> >>>>> `StateStoreMigratedException`
> >>>>>>>>> when
> >>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
> >> we
> >>>>> do not
> >>>>>>>>> need
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
> >>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I understand your point. I rename
> >>>>>>>>> `StreamThreadNotRunningException` to
> >>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> About check unknown state store names:
> >>>>>>>>>>>>>>> Thank you for the hint. I add a new type
> >>>>>>>>> `UnknownStateStoreException`
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> this case.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Also, we should still have fatal exception
> >>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
> >> remove
> >>>>> it?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thank you point this, already add it again.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The KIP already updated, please take a look.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
> Nevertheless, if we omit the categorization, it’s moot.
Ack.
I am fine to remove the middle tier. As John pointed out, it might be
weird to have only one concrete exception type per category. We can also
explain in detail how to handle each exception in their JavaDocs.
-Matthias
On 1/16/20 6:38 AM, Bill Bejeck wrote:
> Vito,
>
> Thanks for the updates, the KIP LGTM.
>
> -Bill
>
> On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org> wrote:
>
>> Hi Vito,
>>
>> Haha, your archive game is on point!
>>
>> What Matthias said in that email is essentially what I figured was the
>> rationale. It makes sense, but the point I was making is that this really
>> doesn’t seem like a good way to structure a production app. On the other
>> hand, considering the exception fatal has a good chance of avoiding a
>> frustrating debug session if you just forgot to call start.
>>
>> Nevertheless, if we omit the categorization, it’s moot.
>>
>> It would be easy to add a categorization layer later if we want it, but
>> not very easy to change it if we get it wrong.
>>
>> Thanks for your consideration!
>> -John
>>
>> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
>>> Hi John,
>>>
>>> About `StreamsNotStartedException is strange` --
>>> The original idea came from Matthias, two years ago. :)
>>> You can reference here:
>>>
>> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>>>
>>> About omitting the categorization --
>>> It looks reasonable. I'm fine with omitting the categorization but not
>> very
>>> sure it is a good choice.
>>> Does any other folks provide opinion?
>>>
>>>
>>> Hi, folks,
>>>
>>> Just update the KIP-216, please take a look.
>>>
>>> ---
>>> Vito
>>>
>>>
>>> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>>>
>>>>
>>>> Hi, folks,
>>>>
>>>> Thank you suggestion, really appreciate it. :)
>>>> I understand your concern. I'll merge StreamsNotRunningException and
>>>> StateStoreNotAvailableException.
>>>>
>>>>
>>>> ---
>>>> Vito
>>>>
>>>>
>>>> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org>
>> wrote:
>>>>
>>>>> Hey Vito,
>>>>>
>>>>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>>>>> Hopefully, we can close it out soon.
>>>>>
>>>>> I agree we can drop StreamsNotRunningException in favor of
>>>>> just StateStoreNotAvailableException.
>>>>>
>>>>> Unfortunately, I have some higher-level concerns. The value
>>>>> of these exceptions is that they tell you how to handle the
>>>>> various situations that can arise while querying a distributed
>>>>> data store.
>>>>>
>>>>> Ideally, as a caller, I should be able to just catch "retriable" or
>>>>> "fatal" and handle them appropriately. Otherwise, there's no
>>>>> point in having categories, and we should just have all the
>>>>> exceptions extend InvalidStateStoreException.
>>>>>
>>>>> Presently, it's not possible to tell from just the
>>>>> "retriable"/"fatal" distinction what to do. You can tell
>>>>> from the descriptions of the various exceptions. E.g.:
>>>>>
>>>>> Retriable:
>>>>> * StreamsRebalancingException: the exact same call
>>>>> should just be retried until the rebalance is complete
>>>>> * StateStoreMigratedException: the store handle is
>>>>> now invalid, so you need to re-discover the instance
>>>>> and get a new handle on that instance. In other words,
>>>>> the query itself may be valid, but the particular method
>>>>> invocation on this particular instance has encountered
>>>>> a fatal exception.
>>>>>
>>>>> Fatal:
>>>>> * UnknownStateStoreException: this is truly fatal. No amount
>>>>> of retrying or re-discovering is going to get you a handle on a
>>>>> store that doesn't exist in the cluster.
>>>>> * StateStoreNotAvailableException: this is actually recoverable,
>>>>> since the store might exist in the cluster, but isn't available on
>>>>> this particular instance (which is shut down or whatever).
>>>>>
>>>>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
>>>>> with omitting the categorization and just having 5 subclasses
>>>>> of InvalidStateStoreException. Each of them would tell you
>>>>> how to handle them, and it's not too many to really
>>>>> understand and handle each one.
>>>>>
>>>>> If you really want to have a middle tier, I'd recommend:
>>>>> * RetryableStateStoreException: the exact same call
>>>>> should be repeated.
>>>>> * RecoverableStateStoreException: the store handle
>>>>> should be discarded and the caller should re-discover
>>>>> the location of the store and repeat the query on the
>>>>> correct instance.
>>>>> * FatalStateStoreException: the query/request is totally
>>>>> invalid and will never succeed.
>>>>>
>>>>> However, attempting to categorize the proposed exceptions
>>>>> reveals even problems with this categorization:
>>>>> Retriable:
>>>>> * StreamsRebalancingException
>>>>> Recoverable:
>>>>> * StateStoreMigratedException
>>>>> * StreamsNotRunningException
>>>>> Fatal:
>>>>> * UnknownStateStoreException
>>>>>
>>>>> But StreamsNotStartedException is strange... It means that
>>>>> one code path got a handle on a specific KafkaStreams object
>>>>> instance and sent it a query before another code path
>>>>> invoked the start() method on the exact same object instance.
>>>>> It seems like the most likely scenario is that whoever wrote
>>>>> the program just forgot to call start() before querying, in
>>>>> which case, retrying isn't going to help, and a fatal exception
>>>>> is more appropriate. I.e., it sounds like a "first 15 minutes
>>>>> experience" problem, and making it fatal would be more
>>>>> helpful. Even in a production context, there's no reason not
>>>>> to sequence your application startup such that you don't
>>>>> accept queries until after Streams is started. Thus, I guess
>>>>> I'd categorize it under "fatal".
>>>>>
>>>>> Regardless of whether you make it fatal or retriable, you'd
>>>>> still have a whole category with only one exception in it,
>>>>> and the other two categories only have two exceptions.
>>>>> Plus, as you pointed out in the KIP, you can't get all
>>>>> exceptions in all cases anyway:
>>>>> * store() can only throw NotStarted, NotRunning,
>>>>> and Unknown
>>>>> * actual store queries can only throw Rebalancing,
>>>>> Migrated, and NotRunning
>>>>>
>>>>> Thus, in practice also, there are exactly three categories
>>>>> and also exactly three exception types. It doesn't seem
>>>>> like there's a great advantage to the categories here. To
>>>>> avoid the categorization problem and also to clarify what
>>>>> exceptions can actually be thrown in different circumstances,
>>>>> it seems like we should just:
>>>>> * get rid of the middle tier and make all the exceptions
>>>>> extend InvalidStateStoreException
>>>>> * drop StateStoreNotAvailableException in favor of
>>>>> StreamsNotRunningException
>>>>> * clearly document on all public methods which exceptions
>>>>> need to be handled
>>>>>
>>>>> How do you feel about this?
>>>>> Thanks,
>>>>> -John
>>>>>
>>>>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
>>>>>> Thanks for KIP Vito.
>>>>>>
>>>>>> Overall the KIP LGTM, but I'd have to agree with others on merging
>> the
>>>>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`
>>>>> classes.
>>>>>>
>>>>>> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
>>>>>> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
>> when to
>>>>>> use the different
>>>>>> exceptions. Maybe a good middle ground would be to have a detailed
>>>>>> exception message.
>>>>>>
>>>>>> The KIP freeze is close, so I think if we can agree on this, we can
>>>>> wrap up
>>>>>> the voting soon.
>>>>>>
>>>>>> Thanks,
>>>>>> Bill
>>>>>>
>>>>>> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Vito,
>>>>>>>
>>>>>>> It's still unclear to me what the advantage is, to have both
>>>>>>> `StreamsNotRunningException` and
>> `StateStoreNotAvailableException`?
>>>>>>>
>>>>>>> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
>> ERROR`
>>>>>>> and thus, for a user point of view, why does it matter if the
>> store is
>>>>>>> closed on not? I don't understand why/how this information would
>> be
>>>>>>> useful? Do you have a concrete example in mind how a user would
>> react
>>>>>>> differently to both exceptions?
>>>>>>>
>>>>>>>
>>>>>>> @Vinoth: about `StreamsRebalancingException` -- to me, it seems
>> best
>>>>> to
>>>>>>> actually do this on a per-query basis, ie, have an overload
>>>>>>> `KafkaStreams#store(...)` that takes a boolean flag that allow to
>>>>>>> _disable_ the exception and opt-in to query a active store during
>>>>>>> recovery. However, as KIP-535 actually introduces this change in
>>>>>>> behavior, I think KIP-216 should not cover this, but KIP-535
>> should be
>>>>>>> updated. I'll follow up on the other KIP thread to raise this
>> point.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 1/11/20 12:26 AM, Vito Jeng wrote:
>>>>>>>> Hi, Matthias & Vinoth,
>>>>>>>>
>>>>>>>> Thanks for the feedback.
>>>>>>>>
>>>>>>>>> What is still unclear to me is, what we gain by having both
>>>>>>>>> `StreamsNotRunningException` and
>>>>> `StateStoreNotAvailableException`. Both
>>>>>>>>> exception are thrown when KafkaStreams is in state
>>>>> PENDING_SHUTDOWN /
>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>> if the
>>>>>>>>> state store is closed on not -- I can't query it anyway? Maybe
>> I
>>>>> miss
>>>>>>>>> something thought?
>>>>>>>>
>>>>>>>> Yes, both `StreamsNotRunningException` and
>>>>>>>> `StateStoreNotAvailableException` are fatal exception.
>>>>>>>> But `StateStoreNotAvailableException` is fatal exception about
>> state
>>>>>>> store
>>>>>>>> related.
>>>>>>>> I think it would be helpful that if user need to distinguish
>> these
>>>>> two
>>>>>>>> different case to handle it.
>>>>>>>>
>>>>>>>> I'm not very sure, does that make sense?
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
>> vinoth@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1 on merging `StreamsNotRunningException` and
>>>>>>>>> `StateStoreNotAvailableException`, both exceptions are fatal
>>>>> anyway. IMO
>>>>>>>>> its best to have these exceptions be about the state store
>> (and not
>>>>>>> streams
>>>>>>>>> state), to easier understanding.
>>>>>>>>>
>>>>>>>>> Additionally, KIP-535 allows for querying of state stores in
>>>>> rebalancing
>>>>>>>>> state. So do we need the StreamsRebalancingException?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2020/01/09 03:38:11, "Matthias J. Sax" <
>> matthias@confluent.io>
>>>>>>> wrote:
>>>>>>>>>> Sorry that I dropped the ball on this...
>>>>>>>>>>
>>>>>>>>>> Thanks for updating the KIP. Overall LGTM now. Feel free to
>> start
>>>>> a
>>>>>>> VOTE
>>>>>>>>>> thread.
>>>>>>>>>>
>>>>>>>>>> What is still unclear to me is, what we gain by having both
>>>>>>>>>> `StreamsNotRunningException` and
>>>>> `StateStoreNotAvailableException`.
>>>>>>> Both
>>>>>>>>>> exception are thrown when KafkaStreams is in state
>>>>> PENDING_SHUTDOWN /
>>>>>>>>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
>> if
>>>>> the
>>>>>>>>>> state store is closed on not -- I can't query it anyway?
>> Maybe I
>>>>> miss
>>>>>>>>>> something thought?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>>>>>>>>>>> Sorry for the late reply, thanks for the review.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> About `StateStoreMigratedException`:
>>>>>>>>>>>>
>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
>>>>> might be
>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
>>>>> back to
>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>> store
>>>>>>> handle.
>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>>>>> `RUNNING`
>>>>>>>>> state.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thank you point this, already updated.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Why do we need to distinguish between
>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> `KafkaStreamsNotRunningException` may be caused by various
>>>>> reasons, I
>>>>>>>>> think
>>>>>>>>>>> it would be helpful that the
>>>>>>>>>>> user can distinguish whether it is caused by the state store
>>>>> closed.
>>>>>>>>>>> (Maybe I am wrong...)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams` instance
>> and
>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
>> the
>>>>>>>>> instance,
>>>>>>>>>>>> because that is the level of granularity in which we
>>>>> enable/disable
>>>>>>>>> IQ atm.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Totally agree. Do you mean the naming of state store
>> exceptions?
>>>>>>>>>>> I don't have special reason to distinguish these two.
>>>>>>>>>>> Your suggestion look more reasonable for the exception
>> naming.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
>> user
>>>>> need
>>>>>>>>> to
>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
>>>>> handle is
>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
>>>>>>> difference
>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>> "blind"
>>>>> retries
>>>>>>>>>>>> that either resolve (if the store is still on the same
>> instance
>>>>> after
>>>>>>>>>>>> rebalancing finishes, or changes to
>>>>> `StateStoreMigratedException` if
>>>>>>>>> the
>>>>>>>>>>>> store was migrated away during rebalancing).
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nice, it's great! Thank you.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The KIP already updated, please take a look. :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
>>>>>>> matthias@confluent.io
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Any update on this KIP?
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>>>>>>>>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
>> busy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> About `StateStoreMigratedException`:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why is it only thrown if the state is REBALANCING? A store
>>>>> might be
>>>>>>>>>>>>> migrated during a rebalance, and Kafka Streams might resume
>>>>> back to
>>>>>>>>>>>>> RUNNING state and afterward somebody tries to use an old
>> store
>>>>>>>>> handle.
>>>>>>>>>>>>> Also, if state is REBALANCING, should we throw
>>>>>>>>>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>>>>>>>>>> `StateStoreMigratedException` does only make sense during
>>>>> `RUNNING`
>>>>>>>>>>>> state.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Why do we need to distinguish between
>>>>>>>>> `KafkaStreamsNotRunningException`
>>>>>>>>>>>>> and `StateStoreNotAvailableException`?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Last, why do we distinguish between `KafkaStreams`
>> instance and
>>>>>>>>>>>>> `StreamsThread`? To me, it seems we should always refer to
>> the
>>>>>>>>> instance,
>>>>>>>>>>>>> because that is the level of granularity in which we
>>>>> enable/disable
>>>>>>>>> IQ
>>>>>>>>>>>> atm.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Last, for `StateStoreMigratedException`, I would add that a
>>>>> user
>>>>>>>>> need to
>>>>>>>>>>>>> rediscover the store and cannot blindly retry as the store
>>>>> handle is
>>>>>>>>>>>>> invalid and a new store handle must be retrieved. That is a
>>>>>>>>> difference
>>>>>>>>>>>>> to `StreamThreadRebalancingException` that allows for
>> "blind"
>>>>>>> retries
>>>>>>>>>>>>> that either resolve (if the store is still on the same
>> instance
>>>>>>> after
>>>>>>>>>>>>> rebalancing finishes, or changes to
>>>>> `StateStoreMigratedException` if
>>>>>>>>> the
>>>>>>>>>>>>> store was migrated away during rebalancing).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>>>>>>>>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9>
>>>>> <https://shorturl.at/CDNT9>
>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>> <https://shorturl.at/CDNT9>
>>>>>>>>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please use the following instead:
>> https://shorturl.at/bkKQU
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
>>>>> vito@is-land.com.tw>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks, Matthias!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About `StreamThreadNotStartedException`:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you for explanation. I agree with your opinion.
>>>>>>>>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>>>>>>>>>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For the case that corresponding thread crashes after we
>>>>> handed out
>>>>>>>>> the
>>>>>>>>>>>>>>> store handle. We may throw
>> `KafkaStreamsNotRunningException`
>>>>> or
>>>>>>>>>>>>>>> `StateStoreMigratedException`.
>>>>>>>>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
>> not
>>>>>>>>> running(
>>>>>>>>>>>>>>> https://shorturl.at/CDNT9) or throw
>>>>> `StateStoreMigratedException`
>>>>>>>>> when
>>>>>>>>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
>> we
>>>>> do not
>>>>>>>>> need
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> add a new type for this case. Does that make sense?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About `KafkaStreamsNotRunningException` vs
>>>>>>>>>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I understand your point. I rename
>>>>>>>>> `StreamThreadNotRunningException` to
>>>>>>>>>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> About check unknown state store names:
>>>>>>>>>>>>>>> Thank you for the hint. I add a new type
>>>>>>>>> `UnknownStateStoreException`
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> this case.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, we should still have fatal exception
>>>>>>>>>>>>>>> `StateStoreNotAvailableException`? Not sure why you
>> remove
>>>>> it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you point this, already add it again.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The KIP already updated, please take a look.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Bill Bejeck <bb...@gmail.com>.
Vito,
Thanks for the updates, the KIP LGTM.
-Bill
On Wed, Jan 15, 2020 at 11:31 PM John Roesler <vv...@apache.org> wrote:
> Hi Vito,
>
> Haha, your archive game is on point!
>
> What Matthias said in that email is essentially what I figured was the
> rationale. It makes sense, but the point I was making is that this really
> doesn’t seem like a good way to structure a production app. On the other
> hand, considering the exception fatal has a good chance of avoiding a
> frustrating debug session if you just forgot to call start.
>
> Nevertheless, if we omit the categorization, it’s moot.
>
> It would be easy to add a categorization layer later if we want it, but
> not very easy to change it if we get it wrong.
>
> Thanks for your consideration!
> -John
>
> On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> > Hi John,
> >
> > About `StreamsNotStartedException is strange` --
> > The original idea came from Matthias, two years ago. :)
> > You can reference here:
> >
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
> >
> > About omitting the categorization --
> > It looks reasonable. I'm fine with omitting the categorization but not
> very
> > sure it is a good choice.
> > Does any other folks provide opinion?
> >
> >
> > Hi, folks,
> >
> > Just update the KIP-216, please take a look.
> >
> > ---
> > Vito
> >
> >
> > On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> >
> > >
> > > Hi, folks,
> > >
> > > Thank you suggestion, really appreciate it. :)
> > > I understand your concern. I'll merge StreamsNotRunningException and
> > > StateStoreNotAvailableException.
> > >
> > >
> > > ---
> > > Vito
> > >
> > >
> > > On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org>
> wrote:
> > >
> > >> Hey Vito,
> > >>
> > >> Yes, thanks for the KIP. Sorry the discussion has been so long.
> > >> Hopefully, we can close it out soon.
> > >>
> > >> I agree we can drop StreamsNotRunningException in favor of
> > >> just StateStoreNotAvailableException.
> > >>
> > >> Unfortunately, I have some higher-level concerns. The value
> > >> of these exceptions is that they tell you how to handle the
> > >> various situations that can arise while querying a distributed
> > >> data store.
> > >>
> > >> Ideally, as a caller, I should be able to just catch "retriable" or
> > >> "fatal" and handle them appropriately. Otherwise, there's no
> > >> point in having categories, and we should just have all the
> > >> exceptions extend InvalidStateStoreException.
> > >>
> > >> Presently, it's not possible to tell from just the
> > >> "retriable"/"fatal" distinction what to do. You can tell
> > >> from the descriptions of the various exceptions. E.g.:
> > >>
> > >> Retriable:
> > >> * StreamsRebalancingException: the exact same call
> > >> should just be retried until the rebalance is complete
> > >> * StateStoreMigratedException: the store handle is
> > >> now invalid, so you need to re-discover the instance
> > >> and get a new handle on that instance. In other words,
> > >> the query itself may be valid, but the particular method
> > >> invocation on this particular instance has encountered
> > >> a fatal exception.
> > >>
> > >> Fatal:
> > >> * UnknownStateStoreException: this is truly fatal. No amount
> > >> of retrying or re-discovering is going to get you a handle on a
> > >> store that doesn't exist in the cluster.
> > >> * StateStoreNotAvailableException: this is actually recoverable,
> > >> since the store might exist in the cluster, but isn't available on
> > >> this particular instance (which is shut down or whatever).
> > >>
> > >> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> > >> with omitting the categorization and just having 5 subclasses
> > >> of InvalidStateStoreException. Each of them would tell you
> > >> how to handle them, and it's not too many to really
> > >> understand and handle each one.
> > >>
> > >> If you really want to have a middle tier, I'd recommend:
> > >> * RetryableStateStoreException: the exact same call
> > >> should be repeated.
> > >> * RecoverableStateStoreException: the store handle
> > >> should be discarded and the caller should re-discover
> > >> the location of the store and repeat the query on the
> > >> correct instance.
> > >> * FatalStateStoreException: the query/request is totally
> > >> invalid and will never succeed.
> > >>
> > >> However, attempting to categorize the proposed exceptions
> > >> reveals even problems with this categorization:
> > >> Retriable:
> > >> * StreamsRebalancingException
> > >> Recoverable:
> > >> * StateStoreMigratedException
> > >> * StreamsNotRunningException
> > >> Fatal:
> > >> * UnknownStateStoreException
> > >>
> > >> But StreamsNotStartedException is strange... It means that
> > >> one code path got a handle on a specific KafkaStreams object
> > >> instance and sent it a query before another code path
> > >> invoked the start() method on the exact same object instance.
> > >> It seems like the most likely scenario is that whoever wrote
> > >> the program just forgot to call start() before querying, in
> > >> which case, retrying isn't going to help, and a fatal exception
> > >> is more appropriate. I.e., it sounds like a "first 15 minutes
> > >> experience" problem, and making it fatal would be more
> > >> helpful. Even in a production context, there's no reason not
> > >> to sequence your application startup such that you don't
> > >> accept queries until after Streams is started. Thus, I guess
> > >> I'd categorize it under "fatal".
> > >>
> > >> Regardless of whether you make it fatal or retriable, you'd
> > >> still have a whole category with only one exception in it,
> > >> and the other two categories only have two exceptions.
> > >> Plus, as you pointed out in the KIP, you can't get all
> > >> exceptions in all cases anyway:
> > >> * store() can only throw NotStarted, NotRunning,
> > >> and Unknown
> > >> * actual store queries can only throw Rebalancing,
> > >> Migrated, and NotRunning
> > >>
> > >> Thus, in practice also, there are exactly three categories
> > >> and also exactly three exception types. It doesn't seem
> > >> like there's a great advantage to the categories here. To
> > >> avoid the categorization problem and also to clarify what
> > >> exceptions can actually be thrown in different circumstances,
> > >> it seems like we should just:
> > >> * get rid of the middle tier and make all the exceptions
> > >> extend InvalidStateStoreException
> > >> * drop StateStoreNotAvailableException in favor of
> > >> StreamsNotRunningException
> > >> * clearly document on all public methods which exceptions
> > >> need to be handled
> > >>
> > >> How do you feel about this?
> > >> Thanks,
> > >> -John
> > >>
> > >> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> > >> > Thanks for KIP Vito.
> > >> >
> > >> > Overall the KIP LGTM, but I'd have to agree with others on merging
> the
> > >> > `StreamsNotRunningException` and `StateStoreNotAvailableException`
> > >> classes.
> > >> >
> > >> > Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> > >> > NOT_RUNNING || ERROR` I'm not even sure how we could distinguish
> when to
> > >> > use the different
> > >> > exceptions. Maybe a good middle ground would be to have a detailed
> > >> > exception message.
> > >> >
> > >> > The KIP freeze is close, so I think if we can agree on this, we can
> > >> wrap up
> > >> > the voting soon.
> > >> >
> > >> > Thanks,
> > >> > Bill
> > >> >
> > >> > On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <
> matthias@confluent.io>
> > >> > wrote:
> > >> >
> > >> > > Vito,
> > >> > >
> > >> > > It's still unclear to me what the advantage is, to have both
> > >> > > `StreamsNotRunningException` and
> `StateStoreNotAvailableException`?
> > >> > >
> > >> > > For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING /
> ERROR`
> > >> > > and thus, for a user point of view, why does it matter if the
> store is
> > >> > > closed on not? I don't understand why/how this information would
> be
> > >> > > useful? Do you have a concrete example in mind how a user would
> react
> > >> > > differently to both exceptions?
> > >> > >
> > >> > >
> > >> > > @Vinoth: about `StreamsRebalancingException` -- to me, it seems
> best
> > >> to
> > >> > > actually do this on a per-query basis, ie, have an overload
> > >> > > `KafkaStreams#store(...)` that takes a boolean flag that allow to
> > >> > > _disable_ the exception and opt-in to query a active store during
> > >> > > recovery. However, as KIP-535 actually introduces this change in
> > >> > > behavior, I think KIP-216 should not cover this, but KIP-535
> should be
> > >> > > updated. I'll follow up on the other KIP thread to raise this
> point.
> > >> > >
> > >> > >
> > >> > > -Matthias
> > >> > >
> > >> > > On 1/11/20 12:26 AM, Vito Jeng wrote:
> > >> > > > Hi, Matthias & Vinoth,
> > >> > > >
> > >> > > > Thanks for the feedback.
> > >> > > >
> > >> > > >> What is still unclear to me is, what we gain by having both
> > >> > > >> `StreamsNotRunningException` and
> > >> `StateStoreNotAvailableException`. Both
> > >> > > >> exception are thrown when KafkaStreams is in state
> > >> PENDING_SHUTDOWN /
> > >> > > >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> if the
> > >> > > >> state store is closed on not -- I can't query it anyway? Maybe
> I
> > >> miss
> > >> > > >> something thought?
> > >> > > >
> > >> > > > Yes, both `StreamsNotRunningException` and
> > >> > > > `StateStoreNotAvailableException` are fatal exception.
> > >> > > > But `StateStoreNotAvailableException` is fatal exception about
> state
> > >> > > store
> > >> > > > related.
> > >> > > > I think it would be helpful that if user need to distinguish
> these
> > >> two
> > >> > > > different case to handle it.
> > >> > > >
> > >> > > > I'm not very sure, does that make sense?
> > >> > > >
> > >> > > >
> > >> > > > ---
> > >> > > > Vito
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <
> vinoth@apache.org>
> > >> > > wrote:
> > >> > > >
> > >> > > >> +1 on merging `StreamsNotRunningException` and
> > >> > > >> `StateStoreNotAvailableException`, both exceptions are fatal
> > >> anyway. IMO
> > >> > > >> its best to have these exceptions be about the state store
> (and not
> > >> > > streams
> > >> > > >> state), to easier understanding.
> > >> > > >>
> > >> > > >> Additionally, KIP-535 allows for querying of state stores in
> > >> rebalancing
> > >> > > >> state. So do we need the StreamsRebalancingException?
> > >> > > >>
> > >> > > >>
> > >> > > >> On 2020/01/09 03:38:11, "Matthias J. Sax" <
> matthias@confluent.io>
> > >> > > wrote:
> > >> > > >>> Sorry that I dropped the ball on this...
> > >> > > >>>
> > >> > > >>> Thanks for updating the KIP. Overall LGTM now. Feel free to
> start
> > >> a
> > >> > > VOTE
> > >> > > >>> thread.
> > >> > > >>>
> > >> > > >>> What is still unclear to me is, what we gain by having both
> > >> > > >>> `StreamsNotRunningException` and
> > >> `StateStoreNotAvailableException`.
> > >> > > Both
> > >> > > >>> exception are thrown when KafkaStreams is in state
> > >> PENDING_SHUTDOWN /
> > >> > > >>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know
> if
> > >> the
> > >> > > >>> state store is closed on not -- I can't query it anyway?
> Maybe I
> > >> miss
> > >> > > >>> something thought?
> > >> > > >>>
> > >> > > >>>
> > >> > > >>> -Matthias
> > >> > > >>>
> > >> > > >>>
> > >> > > >>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> > >> > > >>>> Sorry for the late reply, thanks for the review.
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>> About `StateStoreMigratedException`:
> > >> > > >>>>>
> > >> > > >>>>> Why is it only thrown if the state is REBALANCING? A store
> > >> might be
> > >> > > >>>>> migrated during a rebalance, and Kafka Streams might resume
> > >> back to
> > >> > > >>>>> RUNNING state and afterward somebody tries to use an old
> store
> > >> > > handle.
> > >> > > >>>>> Also, if state is REBALANCING, should we throw
> > >> > > >>>>> `StreamThreadRebalancingException`? Hence, I think
> > >> > > >>>>> `StateStoreMigratedException` does only make sense during
> > >> `RUNNING`
> > >> > > >> state.
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>> Thank you point this, already updated.
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> Why do we need to distinguish between
> > >> > > `KafkaStreamsNotRunningException`
> > >> > > >>>>> and `StateStoreNotAvailableException`?
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>> `KafkaStreamsNotRunningException` may be caused by various
> > >> reasons, I
> > >> > > >> think
> > >> > > >>>> it would be helpful that the
> > >> > > >>>> user can distinguish whether it is caused by the state store
> > >> closed.
> > >> > > >>>> (Maybe I am wrong...)
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> Last, why do we distinguish between `KafkaStreams` instance
> and
> > >> > > >>>>> `StreamsThread`? To me, it seems we should always refer to
> the
> > >> > > >> instance,
> > >> > > >>>>> because that is the level of granularity in which we
> > >> enable/disable
> > >> > > >> IQ atm.
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>> Totally agree. Do you mean the naming of state store
> exceptions?
> > >> > > >>>> I don't have special reason to distinguish these two.
> > >> > > >>>> Your suggestion look more reasonable for the exception
> naming.
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> Last, for `StateStoreMigratedException`, I would add that a
> user
> > >> need
> > >> > > >> to
> > >> > > >>>>> rediscover the store and cannot blindly retry as the store
> > >> handle is
> > >> > > >>>>> invalid and a new store handle must be retrieved. That is a
> > >> > > difference
> > >> > > >>>>> to `StreamThreadRebalancingException` that allows for
> "blind"
> > >> retries
> > >> > > >>>>> that either resolve (if the store is still on the same
> instance
> > >> after
> > >> > > >>>>> rebalancing finishes, or changes to
> > >> `StateStoreMigratedException` if
> > >> > > >> the
> > >> > > >>>>> store was migrated away during rebalancing).
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>> Nice, it's great! Thank you.
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> The KIP already updated, please take a look. :)
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>>
> > >> > > >>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> > >> > > matthias@confluent.io
> > >> > > >>>
> > >> > > >>>> wrote:
> > >> > > >>>>
> > >> > > >>>>> Any update on this KIP?
> > >> > > >>>>>
> > >> > > >>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > >> > > >>>>>> Sorry for the late reply. The 2.4 deadline kept us quite
> busy.
> > >> > > >>>>>>
> > >> > > >>>>>> About `StateStoreMigratedException`:
> > >> > > >>>>>>
> > >> > > >>>>>> Why is it only thrown if the state is REBALANCING? A store
> > >> might be
> > >> > > >>>>>> migrated during a rebalance, and Kafka Streams might resume
> > >> back to
> > >> > > >>>>>> RUNNING state and afterward somebody tries to use an old
> store
> > >> > > >> handle.
> > >> > > >>>>>> Also, if state is REBALANCING, should we throw
> > >> > > >>>>>> `StreamThreadRebalancingException`? Hence, I think
> > >> > > >>>>>> `StateStoreMigratedException` does only make sense during
> > >> `RUNNING`
> > >> > > >>>>> state.
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>> Why do we need to distinguish between
> > >> > > >> `KafkaStreamsNotRunningException`
> > >> > > >>>>>> and `StateStoreNotAvailableException`?
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>> Last, why do we distinguish between `KafkaStreams`
> instance and
> > >> > > >>>>>> `StreamsThread`? To me, it seems we should always refer to
> the
> > >> > > >> instance,
> > >> > > >>>>>> because that is the level of granularity in which we
> > >> enable/disable
> > >> > > >> IQ
> > >> > > >>>>> atm.
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>> Last, for `StateStoreMigratedException`, I would add that a
> > >> user
> > >> > > >> need to
> > >> > > >>>>>> rediscover the store and cannot blindly retry as the store
> > >> handle is
> > >> > > >>>>>> invalid and a new store handle must be retrieved. That is a
> > >> > > >> difference
> > >> > > >>>>>> to `StreamThreadRebalancingException` that allows for
> "blind"
> > >> > > retries
> > >> > > >>>>>> that either resolve (if the store is still on the same
> instance
> > >> > > after
> > >> > > >>>>>> rebalancing finishes, or changes to
> > >> `StateStoreMigratedException` if
> > >> > > >> the
> > >> > > >>>>>> store was migrated away during rebalancing).
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>> -Matthias
> > >> > > >>>>>>
> > >> > > >>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> > >> > > >>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> > >> <https://shorturl.at/CDNT9>
> > >> > > <https://shorturl.at/CDNT9>
> > >> > > >> <https://shorturl.at/CDNT9>
> > >> > > >>>>> <https://shorturl.at/CDNT9>
> > >> > > >>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> > >> > > >>>>>>>
> > >> > > >>>>>>> Please use the following instead:
> https://shorturl.at/bkKQU
> > >> > > >>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>> ---
> > >> > > >>>>>>> Vito
> > >> > > >>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
> > >> vito@is-land.com.tw>
> > >> > > >> wrote:
> > >> > > >>>>>>>
> > >> > > >>>>>>>> Thanks, Matthias!
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>> About `StreamThreadNotStartedException`:
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Thank you for explanation. I agree with your opinion.
> > >> > > >>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> > >> > > >>>>>>>> `StreamThreadNotStartedException`.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> For the case that corresponding thread crashes after we
> > >> handed out
> > >> > > >> the
> > >> > > >>>>>>>> store handle. We may throw
> `KafkaStreamsNotRunningException`
> > >> or
> > >> > > >>>>>>>> `StateStoreMigratedException`.
> > >> > > >>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> > >> > > >>>>>>>> `KafkaStreamsNotRunningException` when stream thread is
> not
> > >> > > >> running(
> > >> > > >>>>>>>> https://shorturl.at/CDNT9) or throw
> > >> `StateStoreMigratedException`
> > >> > > >> when
> > >> > > >>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think
> we
> > >> do not
> > >> > > >> need
> > >> > > >>>>> to
> > >> > > >>>>>>>> add a new type for this case. Does that make sense?
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>> About `KafkaStreamsNotRunningException` vs
> > >> > > >>>>>>>> `StreamThreadNotRunningException`:
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> I understand your point. I rename
> > >> > > >> `StreamThreadNotRunningException` to
> > >> > > >>>>>>>> `KafkaStreamsNotRunningException`.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> About check unknown state store names:
> > >> > > >>>>>>>> Thank you for the hint. I add a new type
> > >> > > >> `UnknownStateStoreException`
> > >> > > >>>>> for
> > >> > > >>>>>>>> this case.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>
> > >> > > >>>>>>>>> Also, we should still have fatal exception
> > >> > > >>>>>>>> `StateStoreNotAvailableException`? Not sure why you
> remove
> > >> it?
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> Thank you point this, already add it again.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> The KIP already updated, please take a look.
> > >> > > >>>>>>>>
> > >> > > >>>>>>>> ---
> > >> > > >>>>>>>> Vito
> > >> > > >>>>>>>>
> > >> > > >>>>>>>
> > >> > > >>>>>>
> > >> > > >>>>>
> > >> > > >>>>>
> > >> > > >>>>
> > >> > > >>>
> > >> > > >>>
> > >> > > >>
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors
Posted by John Roesler <vv...@apache.org>.
Hi Vito,
Haha, your archive game is on point!
What Matthias said in that email is essentially what I figured was the rationale. It makes sense, but the point I was making is that this really doesn’t seem like a good way to structure a production app. On the other hand, considering the exception fatal has a good chance of avoiding a frustrating debug session if you just forgot to call start.
Nevertheless, if we omit the categorization, it’s moot.
It would be easy to add a categorization layer later if we want it, but not very easy to change it if we get it wrong.
Thanks for your consideration!
-John
On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> Hi John,
>
> About `StreamsNotStartedException is strange` --
> The original idea came from Matthias, two years ago. :)
> You can reference here:
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
>
> About omitting the categorization --
> It looks reasonable. I'm fine with omitting the categorization but not very
> sure it is a good choice.
> Does any other folks provide opinion?
>
>
> Hi, folks,
>
> Just update the KIP-216, please take a look.
>
> ---
> Vito
>
>
> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>
> >
> > Hi, folks,
> >
> > Thank you suggestion, really appreciate it. :)
> > I understand your concern. I'll merge StreamsNotRunningException and
> > StateStoreNotAvailableException.
> >
> >
> > ---
> > Vito
> >
> >
> > On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org> wrote:
> >
> >> Hey Vito,
> >>
> >> Yes, thanks for the KIP. Sorry the discussion has been so long.
> >> Hopefully, we can close it out soon.
> >>
> >> I agree we can drop StreamsNotRunningException in favor of
> >> just StateStoreNotAvailableException.
> >>
> >> Unfortunately, I have some higher-level concerns. The value
> >> of these exceptions is that they tell you how to handle the
> >> various situations that can arise while querying a distributed
> >> data store.
> >>
> >> Ideally, as a caller, I should be able to just catch "retriable" or
> >> "fatal" and handle them appropriately. Otherwise, there's no
> >> point in having categories, and we should just have all the
> >> exceptions extend InvalidStateStoreException.
> >>
> >> Presently, it's not possible to tell from just the
> >> "retriable"/"fatal" distinction what to do. You can tell
> >> from the descriptions of the various exceptions. E.g.:
> >>
> >> Retriable:
> >> * StreamsRebalancingException: the exact same call
> >> should just be retried until the rebalance is complete
> >> * StateStoreMigratedException: the store handle is
> >> now invalid, so you need to re-discover the instance
> >> and get a new handle on that instance. In other words,
> >> the query itself may be valid, but the particular method
> >> invocation on this particular instance has encountered
> >> a fatal exception.
> >>
> >> Fatal:
> >> * UnknownStateStoreException: this is truly fatal. No amount
> >> of retrying or re-discovering is going to get you a handle on a
> >> store that doesn't exist in the cluster.
> >> * StateStoreNotAvailableException: this is actually recoverable,
> >> since the store might exist in the cluster, but isn't available on
> >> this particular instance (which is shut down or whatever).
> >>
> >> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> >> with omitting the categorization and just having 5 subclasses
> >> of InvalidStateStoreException. Each of them would tell you
> >> how to handle them, and it's not too many to really
> >> understand and handle each one.
> >>
> >> If you really want to have a middle tier, I'd recommend:
> >> * RetryableStateStoreException: the exact same call
> >> should be repeated.
> >> * RecoverableStateStoreException: the store handle
> >> should be discarded and the caller should re-discover
> >> the location of the store and repeat the query on the
> >> correct instance.
> >> * FatalStateStoreException: the query/request is totally
> >> invalid and will never succeed.
> >>
> >> However, attempting to categorize the proposed exceptions
> >> reveals even problems with this categorization:
> >> Retriable:
> >> * StreamsRebalancingException
> >> Recoverable:
> >> * StateStoreMigratedException
> >> * StreamsNotRunningException
> >> Fatal:
> >> * UnknownStateStoreException
> >>
> >> But StreamsNotStartedException is strange... It means that
> >> one code path got a handle on a specific KafkaStreams object
> >> instance and sent it a query before another code path
> >> invoked the start() method on the exact same object instance.
> >> It seems like the most likely scenario is that whoever wrote
> >> the program just forgot to call start() before querying, in
> >> which case, retrying isn't going to help, and a fatal exception
> >> is more appropriate. I.e., it sounds like a "first 15 minutes
> >> experience" problem, and making it fatal would be more
> >> helpful. Even in a production context, there's no reason not
> >> to sequence your application startup such that you don't
> >> accept queries until after Streams is started. Thus, I guess
> >> I'd categorize it under "fatal".
> >>
> >> Regardless of whether you make it fatal or retriable, you'd
> >> still have a whole category with only one exception in it,
> >> and the other two categories only have two exceptions.
> >> Plus, as you pointed out in the KIP, you can't get all
> >> exceptions in all cases anyway:
> >> * store() can only throw NotStarted, NotRunning,
> >> and Unknown
> >> * actual store queries can only throw Rebalancing,
> >> Migrated, and NotRunning
> >>
> >> Thus, in practice also, there are exactly three categories
> >> and also exactly three exception types. It doesn't seem
> >> like there's a great advantage to the categories here. To
> >> avoid the categorization problem and also to clarify what
> >> exceptions can actually be thrown in different circumstances,
> >> it seems like we should just:
> >> * get rid of the middle tier and make all the exceptions
> >> extend InvalidStateStoreException
> >> * drop StateStoreNotAvailableException in favor of
> >> StreamsNotRunningException
> >> * clearly document on all public methods which exceptions
> >> need to be handled
> >>
> >> How do you feel about this?
> >> Thanks,
> >> -John
> >>
> >> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> >> > Thanks for KIP Vito.
> >> >
> >> > Overall the KIP LGTM, but I'd have to agree with others on merging the
> >> > `StreamsNotRunningException` and `StateStoreNotAvailableException`
> >> classes.
> >> >
> >> > Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> >> > NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
> >> > use the different
> >> > exceptions. Maybe a good middle ground would be to have a detailed
> >> > exception message.
> >> >
> >> > The KIP freeze is close, so I think if we can agree on this, we can
> >> wrap up
> >> > the voting soon.
> >> >
> >> > Thanks,
> >> > Bill
> >> >
> >> > On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <ma...@confluent.io>
> >> > wrote:
> >> >
> >> > > Vito,
> >> > >
> >> > > It's still unclear to me what the advantage is, to have both
> >> > > `StreamsNotRunningException` and `StateStoreNotAvailableException`?
> >> > >
> >> > > For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
> >> > > and thus, for a user point of view, why does it matter if the store is
> >> > > closed on not? I don't understand why/how this information would be
> >> > > useful? Do you have a concrete example in mind how a user would react
> >> > > differently to both exceptions?
> >> > >
> >> > >
> >> > > @Vinoth: about `StreamsRebalancingException` -- to me, it seems best
> >> to
> >> > > actually do this on a per-query basis, ie, have an overload
> >> > > `KafkaStreams#store(...)` that takes a boolean flag that allow to
> >> > > _disable_ the exception and opt-in to query a active store during
> >> > > recovery. However, as KIP-535 actually introduces this change in
> >> > > behavior, I think KIP-216 should not cover this, but KIP-535 should be
> >> > > updated. I'll follow up on the other KIP thread to raise this point.
> >> > >
> >> > >
> >> > > -Matthias
> >> > >
> >> > > On 1/11/20 12:26 AM, Vito Jeng wrote:
> >> > > > Hi, Matthias & Vinoth,
> >> > > >
> >> > > > Thanks for the feedback.
> >> > > >
> >> > > >> What is still unclear to me is, what we gain by having both
> >> > > >> `StreamsNotRunningException` and
> >> `StateStoreNotAvailableException`. Both
> >> > > >> exception are thrown when KafkaStreams is in state
> >> PENDING_SHUTDOWN /
> >> > > >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> >> > > >> state store is closed on not -- I can't query it anyway? Maybe I
> >> miss
> >> > > >> something thought?
> >> > > >
> >> > > > Yes, both `StreamsNotRunningException` and
> >> > > > `StateStoreNotAvailableException` are fatal exception.
> >> > > > But `StateStoreNotAvailableException` is fatal exception about state
> >> > > store
> >> > > > related.
> >> > > > I think it would be helpful that if user need to distinguish these
> >> two
> >> > > > different case to handle it.
> >> > > >
> >> > > > I'm not very sure, does that make sense?
> >> > > >
> >> > > >
> >> > > > ---
> >> > > > Vito
> >> > > >
> >> > > >
> >> > > > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <vi...@apache.org>
> >> > > wrote:
> >> > > >
> >> > > >> +1 on merging `StreamsNotRunningException` and
> >> > > >> `StateStoreNotAvailableException`, both exceptions are fatal
> >> anyway. IMO
> >> > > >> its best to have these exceptions be about the state store (and not
> >> > > streams
> >> > > >> state), to easier understanding.
> >> > > >>
> >> > > >> Additionally, KIP-535 allows for querying of state stores in
> >> rebalancing
> >> > > >> state. So do we need the StreamsRebalancingException?
> >> > > >>
> >> > > >>
> >> > > >> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io>
> >> > > wrote:
> >> > > >>> Sorry that I dropped the ball on this...
> >> > > >>>
> >> > > >>> Thanks for updating the KIP. Overall LGTM now. Feel free to start
> >> a
> >> > > VOTE
> >> > > >>> thread.
> >> > > >>>
> >> > > >>> What is still unclear to me is, what we gain by having both
> >> > > >>> `StreamsNotRunningException` and
> >> `StateStoreNotAvailableException`.
> >> > > Both
> >> > > >>> exception are thrown when KafkaStreams is in state
> >> PENDING_SHUTDOWN /
> >> > > >>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if
> >> the
> >> > > >>> state store is closed on not -- I can't query it anyway? Maybe I
> >> miss
> >> > > >>> something thought?
> >> > > >>>
> >> > > >>>
> >> > > >>> -Matthias
> >> > > >>>
> >> > > >>>
> >> > > >>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> >> > > >>>> Sorry for the late reply, thanks for the review.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>> About `StateStoreMigratedException`:
> >> > > >>>>>
> >> > > >>>>> Why is it only thrown if the state is REBALANCING? A store
> >> might be
> >> > > >>>>> migrated during a rebalance, and Kafka Streams might resume
> >> back to
> >> > > >>>>> RUNNING state and afterward somebody tries to use an old store
> >> > > handle.
> >> > > >>>>> Also, if state is REBALANCING, should we throw
> >> > > >>>>> `StreamThreadRebalancingException`? Hence, I think
> >> > > >>>>> `StateStoreMigratedException` does only make sense during
> >> `RUNNING`
> >> > > >> state.
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>> Thank you point this, already updated.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> Why do we need to distinguish between
> >> > > `KafkaStreamsNotRunningException`
> >> > > >>>>> and `StateStoreNotAvailableException`?
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>> `KafkaStreamsNotRunningException` may be caused by various
> >> reasons, I
> >> > > >> think
> >> > > >>>> it would be helpful that the
> >> > > >>>> user can distinguish whether it is caused by the state store
> >> closed.
> >> > > >>>> (Maybe I am wrong...)
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> Last, why do we distinguish between `KafkaStreams` instance and
> >> > > >>>>> `StreamsThread`? To me, it seems we should always refer to the
> >> > > >> instance,
> >> > > >>>>> because that is the level of granularity in which we
> >> enable/disable
> >> > > >> IQ atm.
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>> Totally agree. Do you mean the naming of state store exceptions?
> >> > > >>>> I don't have special reason to distinguish these two.
> >> > > >>>> Your suggestion look more reasonable for the exception naming.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> Last, for `StateStoreMigratedException`, I would add that a user
> >> need
> >> > > >> to
> >> > > >>>>> rediscover the store and cannot blindly retry as the store
> >> handle is
> >> > > >>>>> invalid and a new store handle must be retrieved. That is a
> >> > > difference
> >> > > >>>>> to `StreamThreadRebalancingException` that allows for "blind"
> >> retries
> >> > > >>>>> that either resolve (if the store is still on the same instance
> >> after
> >> > > >>>>> rebalancing finishes, or changes to
> >> `StateStoreMigratedException` if
> >> > > >> the
> >> > > >>>>> store was migrated away during rebalancing).
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>> Nice, it's great! Thank you.
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> The KIP already updated, please take a look. :)
> >> > > >>>>
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> >> > > matthias@confluent.io
> >> > > >>>
> >> > > >>>> wrote:
> >> > > >>>>
> >> > > >>>>> Any update on this KIP?
> >> > > >>>>>
> >> > > >>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >> > > >>>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> >> > > >>>>>>
> >> > > >>>>>> About `StateStoreMigratedException`:
> >> > > >>>>>>
> >> > > >>>>>> Why is it only thrown if the state is REBALANCING? A store
> >> might be
> >> > > >>>>>> migrated during a rebalance, and Kafka Streams might resume
> >> back to
> >> > > >>>>>> RUNNING state and afterward somebody tries to use an old store
> >> > > >> handle.
> >> > > >>>>>> Also, if state is REBALANCING, should we throw
> >> > > >>>>>> `StreamThreadRebalancingException`? Hence, I think
> >> > > >>>>>> `StateStoreMigratedException` does only make sense during
> >> `RUNNING`
> >> > > >>>>> state.
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> Why do we need to distinguish between
> >> > > >> `KafkaStreamsNotRunningException`
> >> > > >>>>>> and `StateStoreNotAvailableException`?
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> Last, why do we distinguish between `KafkaStreams` instance and
> >> > > >>>>>> `StreamsThread`? To me, it seems we should always refer to the
> >> > > >> instance,
> >> > > >>>>>> because that is the level of granularity in which we
> >> enable/disable
> >> > > >> IQ
> >> > > >>>>> atm.
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> Last, for `StateStoreMigratedException`, I would add that a
> >> user
> >> > > >> need to
> >> > > >>>>>> rediscover the store and cannot blindly retry as the store
> >> handle is
> >> > > >>>>>> invalid and a new store handle must be retrieved. That is a
> >> > > >> difference
> >> > > >>>>>> to `StreamThreadRebalancingException` that allows for "blind"
> >> > > retries
> >> > > >>>>>> that either resolve (if the store is still on the same instance
> >> > > after
> >> > > >>>>>> rebalancing finishes, or changes to
> >> `StateStoreMigratedException` if
> >> > > >> the
> >> > > >>>>>> store was migrated away during rebalancing).
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>>
> >> > > >>>>>> -Matthias
> >> > > >>>>>>
> >> > > >>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >> > > >>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> >> <https://shorturl.at/CDNT9>
> >> > > <https://shorturl.at/CDNT9>
> >> > > >> <https://shorturl.at/CDNT9>
> >> > > >>>>> <https://shorturl.at/CDNT9>
> >> > > >>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> >> > > >>>>>>>
> >> > > >>>>>>> Please use the following instead: https://shorturl.at/bkKQU
> >> > > >>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>> ---
> >> > > >>>>>>> Vito
> >> > > >>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
> >> vito@is-land.com.tw>
> >> > > >> wrote:
> >> > > >>>>>>>
> >> > > >>>>>>>> Thanks, Matthias!
> >> > > >>>>>>>>
> >> > > >>>>>>>>> About `StreamThreadNotStartedException`:
> >> > > >>>>>>>>
> >> > > >>>>>>>> Thank you for explanation. I agree with your opinion.
> >> > > >>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >> > > >>>>>>>> `StreamThreadNotStartedException`.
> >> > > >>>>>>>>
> >> > > >>>>>>>> For the case that corresponding thread crashes after we
> >> handed out
> >> > > >> the
> >> > > >>>>>>>> store handle. We may throw `KafkaStreamsNotRunningException`
> >> or
> >> > > >>>>>>>> `StateStoreMigratedException`.
> >> > > >>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> >> > > >>>>>>>> `KafkaStreamsNotRunningException` when stream thread is not
> >> > > >> running(
> >> > > >>>>>>>> https://shorturl.at/CDNT9) or throw
> >> `StateStoreMigratedException`
> >> > > >> when
> >> > > >>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we
> >> do not
> >> > > >> need
> >> > > >>>>> to
> >> > > >>>>>>>> add a new type for this case. Does that make sense?
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>>> About `KafkaStreamsNotRunningException` vs
> >> > > >>>>>>>> `StreamThreadNotRunningException`:
> >> > > >>>>>>>>
> >> > > >>>>>>>> I understand your point. I rename
> >> > > >> `StreamThreadNotRunningException` to
> >> > > >>>>>>>> `KafkaStreamsNotRunningException`.
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>> About check unknown state store names:
> >> > > >>>>>>>> Thank you for the hint. I add a new type
> >> > > >> `UnknownStateStoreException`
> >> > > >>>>> for
> >> > > >>>>>>>> this case.
> >> > > >>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>>>> Also, we should still have fatal exception
> >> > > >>>>>>>> `StateStoreNotAvailableException`? Not sure why you remove
> >> it?
> >> > > >>>>>>>>
> >> > > >>>>>>>> Thank you point this, already add it again.
> >> > > >>>>>>>>
> >> > > >>>>>>>> The KIP already updated, please take a look.
> >> > > >>>>>>>>
> >> > > >>>>>>>> ---
> >> > > >>>>>>>> Vito
> >> > > >>>>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >>>
> >> > > >>
> >> > > >
> >> > >
> >> > >
> >> >
> >>
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi John,
About `StreamsNotStartedException is strange` --
The original idea came from Matthias, two years ago. :)
You can reference here:
https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc518f@confluent.io%3e
About omitting the categorization --
It looks reasonable. I'm fine with omitting the categorization but not very
sure it is a good choice.
Does any other folks provide opinion?
Hi, folks,
Just update the KIP-216, please take a look.
---
Vito
On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>
> Hi, folks,
>
> Thank you suggestion, really appreciate it. :)
> I understand your concern. I'll merge StreamsNotRunningException and
> StateStoreNotAvailableException.
>
>
> ---
> Vito
>
>
> On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org> wrote:
>
>> Hey Vito,
>>
>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>> Hopefully, we can close it out soon.
>>
>> I agree we can drop StreamsNotRunningException in favor of
>> just StateStoreNotAvailableException.
>>
>> Unfortunately, I have some higher-level concerns. The value
>> of these exceptions is that they tell you how to handle the
>> various situations that can arise while querying a distributed
>> data store.
>>
>> Ideally, as a caller, I should be able to just catch "retriable" or
>> "fatal" and handle them appropriately. Otherwise, there's no
>> point in having categories, and we should just have all the
>> exceptions extend InvalidStateStoreException.
>>
>> Presently, it's not possible to tell from just the
>> "retriable"/"fatal" distinction what to do. You can tell
>> from the descriptions of the various exceptions. E.g.:
>>
>> Retriable:
>> * StreamsRebalancingException: the exact same call
>> should just be retried until the rebalance is complete
>> * StateStoreMigratedException: the store handle is
>> now invalid, so you need to re-discover the instance
>> and get a new handle on that instance. In other words,
>> the query itself may be valid, but the particular method
>> invocation on this particular instance has encountered
>> a fatal exception.
>>
>> Fatal:
>> * UnknownStateStoreException: this is truly fatal. No amount
>> of retrying or re-discovering is going to get you a handle on a
>> store that doesn't exist in the cluster.
>> * StateStoreNotAvailableException: this is actually recoverable,
>> since the store might exist in the cluster, but isn't available on
>> this particular instance (which is shut down or whatever).
>>
>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
>> with omitting the categorization and just having 5 subclasses
>> of InvalidStateStoreException. Each of them would tell you
>> how to handle them, and it's not too many to really
>> understand and handle each one.
>>
>> If you really want to have a middle tier, I'd recommend:
>> * RetryableStateStoreException: the exact same call
>> should be repeated.
>> * RecoverableStateStoreException: the store handle
>> should be discarded and the caller should re-discover
>> the location of the store and repeat the query on the
>> correct instance.
>> * FatalStateStoreException: the query/request is totally
>> invalid and will never succeed.
>>
>> However, attempting to categorize the proposed exceptions
>> reveals even problems with this categorization:
>> Retriable:
>> * StreamsRebalancingException
>> Recoverable:
>> * StateStoreMigratedException
>> * StreamsNotRunningException
>> Fatal:
>> * UnknownStateStoreException
>>
>> But StreamsNotStartedException is strange... It means that
>> one code path got a handle on a specific KafkaStreams object
>> instance and sent it a query before another code path
>> invoked the start() method on the exact same object instance.
>> It seems like the most likely scenario is that whoever wrote
>> the program just forgot to call start() before querying, in
>> which case, retrying isn't going to help, and a fatal exception
>> is more appropriate. I.e., it sounds like a "first 15 minutes
>> experience" problem, and making it fatal would be more
>> helpful. Even in a production context, there's no reason not
>> to sequence your application startup such that you don't
>> accept queries until after Streams is started. Thus, I guess
>> I'd categorize it under "fatal".
>>
>> Regardless of whether you make it fatal or retriable, you'd
>> still have a whole category with only one exception in it,
>> and the other two categories only have two exceptions.
>> Plus, as you pointed out in the KIP, you can't get all
>> exceptions in all cases anyway:
>> * store() can only throw NotStarted, NotRunning,
>> and Unknown
>> * actual store queries can only throw Rebalancing,
>> Migrated, and NotRunning
>>
>> Thus, in practice also, there are exactly three categories
>> and also exactly three exception types. It doesn't seem
>> like there's a great advantage to the categories here. To
>> avoid the categorization problem and also to clarify what
>> exceptions can actually be thrown in different circumstances,
>> it seems like we should just:
>> * get rid of the middle tier and make all the exceptions
>> extend InvalidStateStoreException
>> * drop StateStoreNotAvailableException in favor of
>> StreamsNotRunningException
>> * clearly document on all public methods which exceptions
>> need to be handled
>>
>> How do you feel about this?
>> Thanks,
>> -John
>>
>> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
>> > Thanks for KIP Vito.
>> >
>> > Overall the KIP LGTM, but I'd have to agree with others on merging the
>> > `StreamsNotRunningException` and `StateStoreNotAvailableException`
>> classes.
>> >
>> > Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
>> > NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
>> > use the different
>> > exceptions. Maybe a good middle ground would be to have a detailed
>> > exception message.
>> >
>> > The KIP freeze is close, so I think if we can agree on this, we can
>> wrap up
>> > the voting soon.
>> >
>> > Thanks,
>> > Bill
>> >
>> > On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <ma...@confluent.io>
>> > wrote:
>> >
>> > > Vito,
>> > >
>> > > It's still unclear to me what the advantage is, to have both
>> > > `StreamsNotRunningException` and `StateStoreNotAvailableException`?
>> > >
>> > > For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
>> > > and thus, for a user point of view, why does it matter if the store is
>> > > closed on not? I don't understand why/how this information would be
>> > > useful? Do you have a concrete example in mind how a user would react
>> > > differently to both exceptions?
>> > >
>> > >
>> > > @Vinoth: about `StreamsRebalancingException` -- to me, it seems best
>> to
>> > > actually do this on a per-query basis, ie, have an overload
>> > > `KafkaStreams#store(...)` that takes a boolean flag that allow to
>> > > _disable_ the exception and opt-in to query a active store during
>> > > recovery. However, as KIP-535 actually introduces this change in
>> > > behavior, I think KIP-216 should not cover this, but KIP-535 should be
>> > > updated. I'll follow up on the other KIP thread to raise this point.
>> > >
>> > >
>> > > -Matthias
>> > >
>> > > On 1/11/20 12:26 AM, Vito Jeng wrote:
>> > > > Hi, Matthias & Vinoth,
>> > > >
>> > > > Thanks for the feedback.
>> > > >
>> > > >> What is still unclear to me is, what we gain by having both
>> > > >> `StreamsNotRunningException` and
>> `StateStoreNotAvailableException`. Both
>> > > >> exception are thrown when KafkaStreams is in state
>> PENDING_SHUTDOWN /
>> > > >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
>> > > >> state store is closed on not -- I can't query it anyway? Maybe I
>> miss
>> > > >> something thought?
>> > > >
>> > > > Yes, both `StreamsNotRunningException` and
>> > > > `StateStoreNotAvailableException` are fatal exception.
>> > > > But `StateStoreNotAvailableException` is fatal exception about state
>> > > store
>> > > > related.
>> > > > I think it would be helpful that if user need to distinguish these
>> two
>> > > > different case to handle it.
>> > > >
>> > > > I'm not very sure, does that make sense?
>> > > >
>> > > >
>> > > > ---
>> > > > Vito
>> > > >
>> > > >
>> > > > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <vi...@apache.org>
>> > > wrote:
>> > > >
>> > > >> +1 on merging `StreamsNotRunningException` and
>> > > >> `StateStoreNotAvailableException`, both exceptions are fatal
>> anyway. IMO
>> > > >> its best to have these exceptions be about the state store (and not
>> > > streams
>> > > >> state), to easier understanding.
>> > > >>
>> > > >> Additionally, KIP-535 allows for querying of state stores in
>> rebalancing
>> > > >> state. So do we need the StreamsRebalancingException?
>> > > >>
>> > > >>
>> > > >> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io>
>> > > wrote:
>> > > >>> Sorry that I dropped the ball on this...
>> > > >>>
>> > > >>> Thanks for updating the KIP. Overall LGTM now. Feel free to start
>> a
>> > > VOTE
>> > > >>> thread.
>> > > >>>
>> > > >>> What is still unclear to me is, what we gain by having both
>> > > >>> `StreamsNotRunningException` and
>> `StateStoreNotAvailableException`.
>> > > Both
>> > > >>> exception are thrown when KafkaStreams is in state
>> PENDING_SHUTDOWN /
>> > > >>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if
>> the
>> > > >>> state store is closed on not -- I can't query it anyway? Maybe I
>> miss
>> > > >>> something thought?
>> > > >>>
>> > > >>>
>> > > >>> -Matthias
>> > > >>>
>> > > >>>
>> > > >>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>> > > >>>> Sorry for the late reply, thanks for the review.
>> > > >>>>
>> > > >>>>
>> > > >>>>> About `StateStoreMigratedException`:
>> > > >>>>>
>> > > >>>>> Why is it only thrown if the state is REBALANCING? A store
>> might be
>> > > >>>>> migrated during a rebalance, and Kafka Streams might resume
>> back to
>> > > >>>>> RUNNING state and afterward somebody tries to use an old store
>> > > handle.
>> > > >>>>> Also, if state is REBALANCING, should we throw
>> > > >>>>> `StreamThreadRebalancingException`? Hence, I think
>> > > >>>>> `StateStoreMigratedException` does only make sense during
>> `RUNNING`
>> > > >> state.
>> > > >>>>>
>> > > >>>>
>> > > >>>> Thank you point this, already updated.
>> > > >>>>
>> > > >>>>
>> > > >>>> Why do we need to distinguish between
>> > > `KafkaStreamsNotRunningException`
>> > > >>>>> and `StateStoreNotAvailableException`?
>> > > >>>>>
>> > > >>>>
>> > > >>>> `KafkaStreamsNotRunningException` may be caused by various
>> reasons, I
>> > > >> think
>> > > >>>> it would be helpful that the
>> > > >>>> user can distinguish whether it is caused by the state store
>> closed.
>> > > >>>> (Maybe I am wrong...)
>> > > >>>>
>> > > >>>>
>> > > >>>> Last, why do we distinguish between `KafkaStreams` instance and
>> > > >>>>> `StreamsThread`? To me, it seems we should always refer to the
>> > > >> instance,
>> > > >>>>> because that is the level of granularity in which we
>> enable/disable
>> > > >> IQ atm.
>> > > >>>>>
>> > > >>>>
>> > > >>>> Totally agree. Do you mean the naming of state store exceptions?
>> > > >>>> I don't have special reason to distinguish these two.
>> > > >>>> Your suggestion look more reasonable for the exception naming.
>> > > >>>>
>> > > >>>>
>> > > >>>> Last, for `StateStoreMigratedException`, I would add that a user
>> need
>> > > >> to
>> > > >>>>> rediscover the store and cannot blindly retry as the store
>> handle is
>> > > >>>>> invalid and a new store handle must be retrieved. That is a
>> > > difference
>> > > >>>>> to `StreamThreadRebalancingException` that allows for "blind"
>> retries
>> > > >>>>> that either resolve (if the store is still on the same instance
>> after
>> > > >>>>> rebalancing finishes, or changes to
>> `StateStoreMigratedException` if
>> > > >> the
>> > > >>>>> store was migrated away during rebalancing).
>> > > >>>>>
>> > > >>>>
>> > > >>>> Nice, it's great! Thank you.
>> > > >>>>
>> > > >>>>
>> > > >>>> The KIP already updated, please take a look. :)
>> > > >>>>
>> > > >>>>
>> > > >>>>
>> > > >>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
>> > > matthias@confluent.io
>> > > >>>
>> > > >>>> wrote:
>> > > >>>>
>> > > >>>>> Any update on this KIP?
>> > > >>>>>
>> > > >>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>> > > >>>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
>> > > >>>>>>
>> > > >>>>>> About `StateStoreMigratedException`:
>> > > >>>>>>
>> > > >>>>>> Why is it only thrown if the state is REBALANCING? A store
>> might be
>> > > >>>>>> migrated during a rebalance, and Kafka Streams might resume
>> back to
>> > > >>>>>> RUNNING state and afterward somebody tries to use an old store
>> > > >> handle.
>> > > >>>>>> Also, if state is REBALANCING, should we throw
>> > > >>>>>> `StreamThreadRebalancingException`? Hence, I think
>> > > >>>>>> `StateStoreMigratedException` does only make sense during
>> `RUNNING`
>> > > >>>>> state.
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> Why do we need to distinguish between
>> > > >> `KafkaStreamsNotRunningException`
>> > > >>>>>> and `StateStoreNotAvailableException`?
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> Last, why do we distinguish between `KafkaStreams` instance and
>> > > >>>>>> `StreamsThread`? To me, it seems we should always refer to the
>> > > >> instance,
>> > > >>>>>> because that is the level of granularity in which we
>> enable/disable
>> > > >> IQ
>> > > >>>>> atm.
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> Last, for `StateStoreMigratedException`, I would add that a
>> user
>> > > >> need to
>> > > >>>>>> rediscover the store and cannot blindly retry as the store
>> handle is
>> > > >>>>>> invalid and a new store handle must be retrieved. That is a
>> > > >> difference
>> > > >>>>>> to `StreamThreadRebalancingException` that allows for "blind"
>> > > retries
>> > > >>>>>> that either resolve (if the store is still on the same instance
>> > > after
>> > > >>>>>> rebalancing finishes, or changes to
>> `StateStoreMigratedException` if
>> > > >> the
>> > > >>>>>> store was migrated away during rebalancing).
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>>
>> > > >>>>>> -Matthias
>> > > >>>>>>
>> > > >>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>> > > >>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9>
>> > > <https://shorturl.at/CDNT9>
>> > > >> <https://shorturl.at/CDNT9>
>> > > >>>>> <https://shorturl.at/CDNT9>
>> > > >>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>> > > >>>>>>>
>> > > >>>>>>> Please use the following instead: https://shorturl.at/bkKQU
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> ---
>> > > >>>>>>> Vito
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <
>> vito@is-land.com.tw>
>> > > >> wrote:
>> > > >>>>>>>
>> > > >>>>>>>> Thanks, Matthias!
>> > > >>>>>>>>
>> > > >>>>>>>>> About `StreamThreadNotStartedException`:
>> > > >>>>>>>>
>> > > >>>>>>>> Thank you for explanation. I agree with your opinion.
>> > > >>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>> > > >>>>>>>> `StreamThreadNotStartedException`.
>> > > >>>>>>>>
>> > > >>>>>>>> For the case that corresponding thread crashes after we
>> handed out
>> > > >> the
>> > > >>>>>>>> store handle. We may throw `KafkaStreamsNotRunningException`
>> or
>> > > >>>>>>>> `StateStoreMigratedException`.
>> > > >>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>> > > >>>>>>>> `KafkaStreamsNotRunningException` when stream thread is not
>> > > >> running(
>> > > >>>>>>>> https://shorturl.at/CDNT9) or throw
>> `StateStoreMigratedException`
>> > > >> when
>> > > >>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we
>> do not
>> > > >> need
>> > > >>>>> to
>> > > >>>>>>>> add a new type for this case. Does that make sense?
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>> About `KafkaStreamsNotRunningException` vs
>> > > >>>>>>>> `StreamThreadNotRunningException`:
>> > > >>>>>>>>
>> > > >>>>>>>> I understand your point. I rename
>> > > >> `StreamThreadNotRunningException` to
>> > > >>>>>>>> `KafkaStreamsNotRunningException`.
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>> About check unknown state store names:
>> > > >>>>>>>> Thank you for the hint. I add a new type
>> > > >> `UnknownStateStoreException`
>> > > >>>>> for
>> > > >>>>>>>> this case.
>> > > >>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>>> Also, we should still have fatal exception
>> > > >>>>>>>> `StateStoreNotAvailableException`? Not sure why you remove
>> it?
>> > > >>>>>>>>
>> > > >>>>>>>> Thank you point this, already add it again.
>> > > >>>>>>>>
>> > > >>>>>>>> The KIP already updated, please take a look.
>> > > >>>>>>>>
>> > > >>>>>>>> ---
>> > > >>>>>>>> Vito
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >>>
>> > > >>
>> > > >
>> > >
>> > >
>> >
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, folks,
Thank you suggestion, really appreciate it. :)
I understand your concern. I'll merge StreamsNotRunningException and
StateStoreNotAvailableException.
---
Vito
On Thu, Jan 16, 2020 at 6:22 AM John Roesler <vv...@apache.org> wrote:
> Hey Vito,
>
> Yes, thanks for the KIP. Sorry the discussion has been so long.
> Hopefully, we can close it out soon.
>
> I agree we can drop StreamsNotRunningException in favor of
> just StateStoreNotAvailableException.
>
> Unfortunately, I have some higher-level concerns. The value
> of these exceptions is that they tell you how to handle the
> various situations that can arise while querying a distributed
> data store.
>
> Ideally, as a caller, I should be able to just catch "retriable" or
> "fatal" and handle them appropriately. Otherwise, there's no
> point in having categories, and we should just have all the
> exceptions extend InvalidStateStoreException.
>
> Presently, it's not possible to tell from just the
> "retriable"/"fatal" distinction what to do. You can tell
> from the descriptions of the various exceptions. E.g.:
>
> Retriable:
> * StreamsRebalancingException: the exact same call
> should just be retried until the rebalance is complete
> * StateStoreMigratedException: the store handle is
> now invalid, so you need to re-discover the instance
> and get a new handle on that instance. In other words,
> the query itself may be valid, but the particular method
> invocation on this particular instance has encountered
> a fatal exception.
>
> Fatal:
> * UnknownStateStoreException: this is truly fatal. No amount
> of retrying or re-discovering is going to get you a handle on a
> store that doesn't exist in the cluster.
> * StateStoreNotAvailableException: this is actually recoverable,
> since the store might exist in the cluster, but isn't available on
> this particular instance (which is shut down or whatever).
>
> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> with omitting the categorization and just having 5 subclasses
> of InvalidStateStoreException. Each of them would tell you
> how to handle them, and it's not too many to really
> understand and handle each one.
>
> If you really want to have a middle tier, I'd recommend:
> * RetryableStateStoreException: the exact same call
> should be repeated.
> * RecoverableStateStoreException: the store handle
> should be discarded and the caller should re-discover
> the location of the store and repeat the query on the
> correct instance.
> * FatalStateStoreException: the query/request is totally
> invalid and will never succeed.
>
> However, attempting to categorize the proposed exceptions
> reveals even problems with this categorization:
> Retriable:
> * StreamsRebalancingException
> Recoverable:
> * StateStoreMigratedException
> * StreamsNotRunningException
> Fatal:
> * UnknownStateStoreException
>
> But StreamsNotStartedException is strange... It means that
> one code path got a handle on a specific KafkaStreams object
> instance and sent it a query before another code path
> invoked the start() method on the exact same object instance.
> It seems like the most likely scenario is that whoever wrote
> the program just forgot to call start() before querying, in
> which case, retrying isn't going to help, and a fatal exception
> is more appropriate. I.e., it sounds like a "first 15 minutes
> experience" problem, and making it fatal would be more
> helpful. Even in a production context, there's no reason not
> to sequence your application startup such that you don't
> accept queries until after Streams is started. Thus, I guess
> I'd categorize it under "fatal".
>
> Regardless of whether you make it fatal or retriable, you'd
> still have a whole category with only one exception in it,
> and the other two categories only have two exceptions.
> Plus, as you pointed out in the KIP, you can't get all
> exceptions in all cases anyway:
> * store() can only throw NotStarted, NotRunning,
> and Unknown
> * actual store queries can only throw Rebalancing,
> Migrated, and NotRunning
>
> Thus, in practice also, there are exactly three categories
> and also exactly three exception types. It doesn't seem
> like there's a great advantage to the categories here. To
> avoid the categorization problem and also to clarify what
> exceptions can actually be thrown in different circumstances,
> it seems like we should just:
> * get rid of the middle tier and make all the exceptions
> extend InvalidStateStoreException
> * drop StateStoreNotAvailableException in favor of
> StreamsNotRunningException
> * clearly document on all public methods which exceptions
> need to be handled
>
> How do you feel about this?
> Thanks,
> -John
>
> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> > Thanks for KIP Vito.
> >
> > Overall the KIP LGTM, but I'd have to agree with others on merging the
> > `StreamsNotRunningException` and `StateStoreNotAvailableException`
> classes.
> >
> > Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> > NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
> > use the different
> > exceptions. Maybe a good middle ground would be to have a detailed
> > exception message.
> >
> > The KIP freeze is close, so I think if we can agree on this, we can wrap
> up
> > the voting soon.
> >
> > Thanks,
> > Bill
> >
> > On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> > > Vito,
> > >
> > > It's still unclear to me what the advantage is, to have both
> > > `StreamsNotRunningException` and `StateStoreNotAvailableException`?
> > >
> > > For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
> > > and thus, for a user point of view, why does it matter if the store is
> > > closed on not? I don't understand why/how this information would be
> > > useful? Do you have a concrete example in mind how a user would react
> > > differently to both exceptions?
> > >
> > >
> > > @Vinoth: about `StreamsRebalancingException` -- to me, it seems best to
> > > actually do this on a per-query basis, ie, have an overload
> > > `KafkaStreams#store(...)` that takes a boolean flag that allow to
> > > _disable_ the exception and opt-in to query a active store during
> > > recovery. However, as KIP-535 actually introduces this change in
> > > behavior, I think KIP-216 should not cover this, but KIP-535 should be
> > > updated. I'll follow up on the other KIP thread to raise this point.
> > >
> > >
> > > -Matthias
> > >
> > > On 1/11/20 12:26 AM, Vito Jeng wrote:
> > > > Hi, Matthias & Vinoth,
> > > >
> > > > Thanks for the feedback.
> > > >
> > > >> What is still unclear to me is, what we gain by having both
> > > >> `StreamsNotRunningException` and `StateStoreNotAvailableException`.
> Both
> > > >> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN
> /
> > > >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> > > >> state store is closed on not -- I can't query it anyway? Maybe I
> miss
> > > >> something thought?
> > > >
> > > > Yes, both `StreamsNotRunningException` and
> > > > `StateStoreNotAvailableException` are fatal exception.
> > > > But `StateStoreNotAvailableException` is fatal exception about state
> > > store
> > > > related.
> > > > I think it would be helpful that if user need to distinguish these
> two
> > > > different case to handle it.
> > > >
> > > > I'm not very sure, does that make sense?
> > > >
> > > >
> > > > ---
> > > > Vito
> > > >
> > > >
> > > > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <vi...@apache.org>
> > > wrote:
> > > >
> > > >> +1 on merging `StreamsNotRunningException` and
> > > >> `StateStoreNotAvailableException`, both exceptions are fatal
> anyway. IMO
> > > >> its best to have these exceptions be about the state store (and not
> > > streams
> > > >> state), to easier understanding.
> > > >>
> > > >> Additionally, KIP-535 allows for querying of state stores in
> rebalancing
> > > >> state. So do we need the StreamsRebalancingException?
> > > >>
> > > >>
> > > >> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io>
> > > wrote:
> > > >>> Sorry that I dropped the ball on this...
> > > >>>
> > > >>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a
> > > VOTE
> > > >>> thread.
> > > >>>
> > > >>> What is still unclear to me is, what we gain by having both
> > > >>> `StreamsNotRunningException` and `StateStoreNotAvailableException`.
> > > Both
> > > >>> exception are thrown when KafkaStreams is in state
> PENDING_SHUTDOWN /
> > > >>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> > > >>> state store is closed on not -- I can't query it anyway? Maybe I
> miss
> > > >>> something thought?
> > > >>>
> > > >>>
> > > >>> -Matthias
> > > >>>
> > > >>>
> > > >>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> > > >>>> Sorry for the late reply, thanks for the review.
> > > >>>>
> > > >>>>
> > > >>>>> About `StateStoreMigratedException`:
> > > >>>>>
> > > >>>>> Why is it only thrown if the state is REBALANCING? A store might
> be
> > > >>>>> migrated during a rebalance, and Kafka Streams might resume back
> to
> > > >>>>> RUNNING state and afterward somebody tries to use an old store
> > > handle.
> > > >>>>> Also, if state is REBALANCING, should we throw
> > > >>>>> `StreamThreadRebalancingException`? Hence, I think
> > > >>>>> `StateStoreMigratedException` does only make sense during
> `RUNNING`
> > > >> state.
> > > >>>>>
> > > >>>>
> > > >>>> Thank you point this, already updated.
> > > >>>>
> > > >>>>
> > > >>>> Why do we need to distinguish between
> > > `KafkaStreamsNotRunningException`
> > > >>>>> and `StateStoreNotAvailableException`?
> > > >>>>>
> > > >>>>
> > > >>>> `KafkaStreamsNotRunningException` may be caused by various
> reasons, I
> > > >> think
> > > >>>> it would be helpful that the
> > > >>>> user can distinguish whether it is caused by the state store
> closed.
> > > >>>> (Maybe I am wrong...)
> > > >>>>
> > > >>>>
> > > >>>> Last, why do we distinguish between `KafkaStreams` instance and
> > > >>>>> `StreamsThread`? To me, it seems we should always refer to the
> > > >> instance,
> > > >>>>> because that is the level of granularity in which we
> enable/disable
> > > >> IQ atm.
> > > >>>>>
> > > >>>>
> > > >>>> Totally agree. Do you mean the naming of state store exceptions?
> > > >>>> I don't have special reason to distinguish these two.
> > > >>>> Your suggestion look more reasonable for the exception naming.
> > > >>>>
> > > >>>>
> > > >>>> Last, for `StateStoreMigratedException`, I would add that a user
> need
> > > >> to
> > > >>>>> rediscover the store and cannot blindly retry as the store
> handle is
> > > >>>>> invalid and a new store handle must be retrieved. That is a
> > > difference
> > > >>>>> to `StreamThreadRebalancingException` that allows for "blind"
> retries
> > > >>>>> that either resolve (if the store is still on the same instance
> after
> > > >>>>> rebalancing finishes, or changes to
> `StateStoreMigratedException` if
> > > >> the
> > > >>>>> store was migrated away during rebalancing).
> > > >>>>>
> > > >>>>
> > > >>>> Nice, it's great! Thank you.
> > > >>>>
> > > >>>>
> > > >>>> The KIP already updated, please take a look. :)
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> > > matthias@confluent.io
> > > >>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Any update on this KIP?
> > > >>>>>
> > > >>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > > >>>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> > > >>>>>>
> > > >>>>>> About `StateStoreMigratedException`:
> > > >>>>>>
> > > >>>>>> Why is it only thrown if the state is REBALANCING? A store
> might be
> > > >>>>>> migrated during a rebalance, and Kafka Streams might resume
> back to
> > > >>>>>> RUNNING state and afterward somebody tries to use an old store
> > > >> handle.
> > > >>>>>> Also, if state is REBALANCING, should we throw
> > > >>>>>> `StreamThreadRebalancingException`? Hence, I think
> > > >>>>>> `StateStoreMigratedException` does only make sense during
> `RUNNING`
> > > >>>>> state.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Why do we need to distinguish between
> > > >> `KafkaStreamsNotRunningException`
> > > >>>>>> and `StateStoreNotAvailableException`?
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Last, why do we distinguish between `KafkaStreams` instance and
> > > >>>>>> `StreamsThread`? To me, it seems we should always refer to the
> > > >> instance,
> > > >>>>>> because that is the level of granularity in which we
> enable/disable
> > > >> IQ
> > > >>>>> atm.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Last, for `StateStoreMigratedException`, I would add that a user
> > > >> need to
> > > >>>>>> rediscover the store and cannot blindly retry as the store
> handle is
> > > >>>>>> invalid and a new store handle must be retrieved. That is a
> > > >> difference
> > > >>>>>> to `StreamThreadRebalancingException` that allows for "blind"
> > > retries
> > > >>>>>> that either resolve (if the store is still on the same instance
> > > after
> > > >>>>>> rebalancing finishes, or changes to
> `StateStoreMigratedException` if
> > > >> the
> > > >>>>>> store was migrated away during rebalancing).
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> -Matthias
> > > >>>>>>
> > > >>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> > > >>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> > > <https://shorturl.at/CDNT9>
> > > >> <https://shorturl.at/CDNT9>
> > > >>>>> <https://shorturl.at/CDNT9>
> > > >>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> > > >>>>>>>
> > > >>>>>>> Please use the following instead: https://shorturl.at/bkKQU
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> ---
> > > >>>>>>> Vito
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vito@is-land.com.tw
> >
> > > >> wrote:
> > > >>>>>>>
> > > >>>>>>>> Thanks, Matthias!
> > > >>>>>>>>
> > > >>>>>>>>> About `StreamThreadNotStartedException`:
> > > >>>>>>>>
> > > >>>>>>>> Thank you for explanation. I agree with your opinion.
> > > >>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> > > >>>>>>>> `StreamThreadNotStartedException`.
> > > >>>>>>>>
> > > >>>>>>>> For the case that corresponding thread crashes after we
> handed out
> > > >> the
> > > >>>>>>>> store handle. We may throw `KafkaStreamsNotRunningException`
> or
> > > >>>>>>>> `StateStoreMigratedException`.
> > > >>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> > > >>>>>>>> `KafkaStreamsNotRunningException` when stream thread is not
> > > >> running(
> > > >>>>>>>> https://shorturl.at/CDNT9) or throw
> `StateStoreMigratedException`
> > > >> when
> > > >>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we do
> not
> > > >> need
> > > >>>>> to
> > > >>>>>>>> add a new type for this case. Does that make sense?
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> About `KafkaStreamsNotRunningException` vs
> > > >>>>>>>> `StreamThreadNotRunningException`:
> > > >>>>>>>>
> > > >>>>>>>> I understand your point. I rename
> > > >> `StreamThreadNotRunningException` to
> > > >>>>>>>> `KafkaStreamsNotRunningException`.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> About check unknown state store names:
> > > >>>>>>>> Thank you for the hint. I add a new type
> > > >> `UnknownStateStoreException`
> > > >>>>> for
> > > >>>>>>>> this case.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> Also, we should still have fatal exception
> > > >>>>>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
> > > >>>>>>>>
> > > >>>>>>>> Thank you point this, already add it again.
> > > >>>>>>>>
> > > >>>>>>>> The KIP already updated, please take a look.
> > > >>>>>>>>
> > > >>>>>>>> ---
> > > >>>>>>>> Vito
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors
Posted by John Roesler <vv...@apache.org>.
Hey Vito,
Yes, thanks for the KIP. Sorry the discussion has been so long.
Hopefully, we can close it out soon.
I agree we can drop StreamsNotRunningException in favor of
just StateStoreNotAvailableException.
Unfortunately, I have some higher-level concerns. The value
of these exceptions is that they tell you how to handle the
various situations that can arise while querying a distributed
data store.
Ideally, as a caller, I should be able to just catch "retriable" or
"fatal" and handle them appropriately. Otherwise, there's no
point in having categories, and we should just have all the
exceptions extend InvalidStateStoreException.
Presently, it's not possible to tell from just the
"retriable"/"fatal" distinction what to do. You can tell
from the descriptions of the various exceptions. E.g.:
Retriable:
* StreamsRebalancingException: the exact same call
should just be retried until the rebalance is complete
* StateStoreMigratedException: the store handle is
now invalid, so you need to re-discover the instance
and get a new handle on that instance. In other words,
the query itself may be valid, but the particular method
invocation on this particular instance has encountered
a fatal exception.
Fatal:
* UnknownStateStoreException: this is truly fatal. No amount
of retrying or re-discovering is going to get you a handle on a
store that doesn't exist in the cluster.
* StateStoreNotAvailableException: this is actually recoverable,
since the store might exist in the cluster, but isn't available on
this particular instance (which is shut down or whatever).
Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
with omitting the categorization and just having 5 subclasses
of InvalidStateStoreException. Each of them would tell you
how to handle them, and it's not too many to really
understand and handle each one.
If you really want to have a middle tier, I'd recommend:
* RetryableStateStoreException: the exact same call
should be repeated.
* RecoverableStateStoreException: the store handle
should be discarded and the caller should re-discover
the location of the store and repeat the query on the
correct instance.
* FatalStateStoreException: the query/request is totally
invalid and will never succeed.
However, attempting to categorize the proposed exceptions
reveals even problems with this categorization:
Retriable:
* StreamsRebalancingException
Recoverable:
* StateStoreMigratedException
* StreamsNotRunningException
Fatal:
* UnknownStateStoreException
But StreamsNotStartedException is strange... It means that
one code path got a handle on a specific KafkaStreams object
instance and sent it a query before another code path
invoked the start() method on the exact same object instance.
It seems like the most likely scenario is that whoever wrote
the program just forgot to call start() before querying, in
which case, retrying isn't going to help, and a fatal exception
is more appropriate. I.e., it sounds like a "first 15 minutes
experience" problem, and making it fatal would be more
helpful. Even in a production context, there's no reason not
to sequence your application startup such that you don't
accept queries until after Streams is started. Thus, I guess
I'd categorize it under "fatal".
Regardless of whether you make it fatal or retriable, you'd
still have a whole category with only one exception in it,
and the other two categories only have two exceptions.
Plus, as you pointed out in the KIP, you can't get all
exceptions in all cases anyway:
* store() can only throw NotStarted, NotRunning,
and Unknown
* actual store queries can only throw Rebalancing,
Migrated, and NotRunning
Thus, in practice also, there are exactly three categories
and also exactly three exception types. It doesn't seem
like there's a great advantage to the categories here. To
avoid the categorization problem and also to clarify what
exceptions can actually be thrown in different circumstances,
it seems like we should just:
* get rid of the middle tier and make all the exceptions
extend InvalidStateStoreException
* drop StateStoreNotAvailableException in favor of
StreamsNotRunningException
* clearly document on all public methods which exceptions
need to be handled
How do you feel about this?
Thanks,
-John
On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> Thanks for KIP Vito.
>
> Overall the KIP LGTM, but I'd have to agree with others on merging the
> `StreamsNotRunningException` and `StateStoreNotAvailableException` classes.
>
> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
> use the different
> exceptions. Maybe a good middle ground would be to have a detailed
> exception message.
>
> The KIP freeze is close, so I think if we can agree on this, we can wrap up
> the voting soon.
>
> Thanks,
> Bill
>
> On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Vito,
> >
> > It's still unclear to me what the advantage is, to have both
> > `StreamsNotRunningException` and `StateStoreNotAvailableException`?
> >
> > For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
> > and thus, for a user point of view, why does it matter if the store is
> > closed on not? I don't understand why/how this information would be
> > useful? Do you have a concrete example in mind how a user would react
> > differently to both exceptions?
> >
> >
> > @Vinoth: about `StreamsRebalancingException` -- to me, it seems best to
> > actually do this on a per-query basis, ie, have an overload
> > `KafkaStreams#store(...)` that takes a boolean flag that allow to
> > _disable_ the exception and opt-in to query a active store during
> > recovery. However, as KIP-535 actually introduces this change in
> > behavior, I think KIP-216 should not cover this, but KIP-535 should be
> > updated. I'll follow up on the other KIP thread to raise this point.
> >
> >
> > -Matthias
> >
> > On 1/11/20 12:26 AM, Vito Jeng wrote:
> > > Hi, Matthias & Vinoth,
> > >
> > > Thanks for the feedback.
> > >
> > >> What is still unclear to me is, what we gain by having both
> > >> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> > >> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> > >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> > >> state store is closed on not -- I can't query it anyway? Maybe I miss
> > >> something thought?
> > >
> > > Yes, both `StreamsNotRunningException` and
> > > `StateStoreNotAvailableException` are fatal exception.
> > > But `StateStoreNotAvailableException` is fatal exception about state
> > store
> > > related.
> > > I think it would be helpful that if user need to distinguish these two
> > > different case to handle it.
> > >
> > > I'm not very sure, does that make sense?
> > >
> > >
> > > ---
> > > Vito
> > >
> > >
> > > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > >
> > >> +1 on merging `StreamsNotRunningException` and
> > >> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
> > >> its best to have these exceptions be about the state store (and not
> > streams
> > >> state), to easier understanding.
> > >>
> > >> Additionally, KIP-535 allows for querying of state stores in rebalancing
> > >> state. So do we need the StreamsRebalancingException?
> > >>
> > >>
> > >> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io>
> > wrote:
> > >>> Sorry that I dropped the ball on this...
> > >>>
> > >>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a
> > VOTE
> > >>> thread.
> > >>>
> > >>> What is still unclear to me is, what we gain by having both
> > >>> `StreamsNotRunningException` and `StateStoreNotAvailableException`.
> > Both
> > >>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> > >>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> > >>> state store is closed on not -- I can't query it anyway? Maybe I miss
> > >>> something thought?
> > >>>
> > >>>
> > >>> -Matthias
> > >>>
> > >>>
> > >>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> > >>>> Sorry for the late reply, thanks for the review.
> > >>>>
> > >>>>
> > >>>>> About `StateStoreMigratedException`:
> > >>>>>
> > >>>>> Why is it only thrown if the state is REBALANCING? A store might be
> > >>>>> migrated during a rebalance, and Kafka Streams might resume back to
> > >>>>> RUNNING state and afterward somebody tries to use an old store
> > handle.
> > >>>>> Also, if state is REBALANCING, should we throw
> > >>>>> `StreamThreadRebalancingException`? Hence, I think
> > >>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
> > >> state.
> > >>>>>
> > >>>>
> > >>>> Thank you point this, already updated.
> > >>>>
> > >>>>
> > >>>> Why do we need to distinguish between
> > `KafkaStreamsNotRunningException`
> > >>>>> and `StateStoreNotAvailableException`?
> > >>>>>
> > >>>>
> > >>>> `KafkaStreamsNotRunningException` may be caused by various reasons, I
> > >> think
> > >>>> it would be helpful that the
> > >>>> user can distinguish whether it is caused by the state store closed.
> > >>>> (Maybe I am wrong...)
> > >>>>
> > >>>>
> > >>>> Last, why do we distinguish between `KafkaStreams` instance and
> > >>>>> `StreamsThread`? To me, it seems we should always refer to the
> > >> instance,
> > >>>>> because that is the level of granularity in which we enable/disable
> > >> IQ atm.
> > >>>>>
> > >>>>
> > >>>> Totally agree. Do you mean the naming of state store exceptions?
> > >>>> I don't have special reason to distinguish these two.
> > >>>> Your suggestion look more reasonable for the exception naming.
> > >>>>
> > >>>>
> > >>>> Last, for `StateStoreMigratedException`, I would add that a user need
> > >> to
> > >>>>> rediscover the store and cannot blindly retry as the store handle is
> > >>>>> invalid and a new store handle must be retrieved. That is a
> > difference
> > >>>>> to `StreamThreadRebalancingException` that allows for "blind" retries
> > >>>>> that either resolve (if the store is still on the same instance after
> > >>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
> > >> the
> > >>>>> store was migrated away during rebalancing).
> > >>>>>
> > >>>>
> > >>>> Nice, it's great! Thank you.
> > >>>>
> > >>>>
> > >>>> The KIP already updated, please take a look. :)
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> > matthias@confluent.io
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> Any update on this KIP?
> > >>>>>
> > >>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > >>>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> > >>>>>>
> > >>>>>> About `StateStoreMigratedException`:
> > >>>>>>
> > >>>>>> Why is it only thrown if the state is REBALANCING? A store might be
> > >>>>>> migrated during a rebalance, and Kafka Streams might resume back to
> > >>>>>> RUNNING state and afterward somebody tries to use an old store
> > >> handle.
> > >>>>>> Also, if state is REBALANCING, should we throw
> > >>>>>> `StreamThreadRebalancingException`? Hence, I think
> > >>>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
> > >>>>> state.
> > >>>>>>
> > >>>>>>
> > >>>>>> Why do we need to distinguish between
> > >> `KafkaStreamsNotRunningException`
> > >>>>>> and `StateStoreNotAvailableException`?
> > >>>>>>
> > >>>>>>
> > >>>>>> Last, why do we distinguish between `KafkaStreams` instance and
> > >>>>>> `StreamsThread`? To me, it seems we should always refer to the
> > >> instance,
> > >>>>>> because that is the level of granularity in which we enable/disable
> > >> IQ
> > >>>>> atm.
> > >>>>>>
> > >>>>>>
> > >>>>>> Last, for `StateStoreMigratedException`, I would add that a user
> > >> need to
> > >>>>>> rediscover the store and cannot blindly retry as the store handle is
> > >>>>>> invalid and a new store handle must be retrieved. That is a
> > >> difference
> > >>>>>> to `StreamThreadRebalancingException` that allows for "blind"
> > retries
> > >>>>>> that either resolve (if the store is still on the same instance
> > after
> > >>>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
> > >> the
> > >>>>>> store was migrated away during rebalancing).
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> -Matthias
> > >>>>>>
> > >>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> > >>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> > <https://shorturl.at/CDNT9>
> > >> <https://shorturl.at/CDNT9>
> > >>>>> <https://shorturl.at/CDNT9>
> > >>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> > >>>>>>>
> > >>>>>>> Please use the following instead: https://shorturl.at/bkKQU
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> ---
> > >>>>>>> Vito
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw>
> > >> wrote:
> > >>>>>>>
> > >>>>>>>> Thanks, Matthias!
> > >>>>>>>>
> > >>>>>>>>> About `StreamThreadNotStartedException`:
> > >>>>>>>>
> > >>>>>>>> Thank you for explanation. I agree with your opinion.
> > >>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> > >>>>>>>> `StreamThreadNotStartedException`.
> > >>>>>>>>
> > >>>>>>>> For the case that corresponding thread crashes after we handed out
> > >> the
> > >>>>>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
> > >>>>>>>> `StateStoreMigratedException`.
> > >>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> > >>>>>>>> `KafkaStreamsNotRunningException` when stream thread is not
> > >> running(
> > >>>>>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException`
> > >> when
> > >>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not
> > >> need
> > >>>>> to
> > >>>>>>>> add a new type for this case. Does that make sense?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> About `KafkaStreamsNotRunningException` vs
> > >>>>>>>> `StreamThreadNotRunningException`:
> > >>>>>>>>
> > >>>>>>>> I understand your point. I rename
> > >> `StreamThreadNotRunningException` to
> > >>>>>>>> `KafkaStreamsNotRunningException`.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> About check unknown state store names:
> > >>>>>>>> Thank you for the hint. I add a new type
> > >> `UnknownStateStoreException`
> > >>>>> for
> > >>>>>>>> this case.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> Also, we should still have fatal exception
> > >>>>>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
> > >>>>>>>>
> > >>>>>>>> Thank you point this, already add it again.
> > >>>>>>>>
> > >>>>>>>> The KIP already updated, please take a look.
> > >>>>>>>>
> > >>>>>>>> ---
> > >>>>>>>> Vito
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >
> >
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Bill Bejeck <bb...@gmail.com>.
Thanks for KIP Vito.
Overall the KIP LGTM, but I'd have to agree with others on merging the
`StreamsNotRunningException` and `StateStoreNotAvailableException` classes.
Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
use the different
exceptions. Maybe a good middle ground would be to have a detailed
exception message.
The KIP freeze is close, so I think if we can agree on this, we can wrap up
the voting soon.
Thanks,
Bill
On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax <ma...@confluent.io>
wrote:
> Vito,
>
> It's still unclear to me what the advantage is, to have both
> `StreamsNotRunningException` and `StateStoreNotAvailableException`?
>
> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
> and thus, for a user point of view, why does it matter if the store is
> closed on not? I don't understand why/how this information would be
> useful? Do you have a concrete example in mind how a user would react
> differently to both exceptions?
>
>
> @Vinoth: about `StreamsRebalancingException` -- to me, it seems best to
> actually do this on a per-query basis, ie, have an overload
> `KafkaStreams#store(...)` that takes a boolean flag that allow to
> _disable_ the exception and opt-in to query a active store during
> recovery. However, as KIP-535 actually introduces this change in
> behavior, I think KIP-216 should not cover this, but KIP-535 should be
> updated. I'll follow up on the other KIP thread to raise this point.
>
>
> -Matthias
>
> On 1/11/20 12:26 AM, Vito Jeng wrote:
> > Hi, Matthias & Vinoth,
> >
> > Thanks for the feedback.
> >
> >> What is still unclear to me is, what we gain by having both
> >> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> >> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> >> state store is closed on not -- I can't query it anyway? Maybe I miss
> >> something thought?
> >
> > Yes, both `StreamsNotRunningException` and
> > `StateStoreNotAvailableException` are fatal exception.
> > But `StateStoreNotAvailableException` is fatal exception about state
> store
> > related.
> > I think it would be helpful that if user need to distinguish these two
> > different case to handle it.
> >
> > I'm not very sure, does that make sense?
> >
> >
> > ---
> > Vito
> >
> >
> > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <vi...@apache.org>
> wrote:
> >
> >> +1 on merging `StreamsNotRunningException` and
> >> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
> >> its best to have these exceptions be about the state store (and not
> streams
> >> state), to easier understanding.
> >>
> >> Additionally, KIP-535 allows for querying of state stores in rebalancing
> >> state. So do we need the StreamsRebalancingException?
> >>
> >>
> >> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io>
> wrote:
> >>> Sorry that I dropped the ball on this...
> >>>
> >>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a
> VOTE
> >>> thread.
> >>>
> >>> What is still unclear to me is, what we gain by having both
> >>> `StreamsNotRunningException` and `StateStoreNotAvailableException`.
> Both
> >>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> >>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> >>> state store is closed on not -- I can't query it anyway? Maybe I miss
> >>> something thought?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>>
> >>> On 11/3/19 6:07 PM, Vito Jeng wrote:
> >>>> Sorry for the late reply, thanks for the review.
> >>>>
> >>>>
> >>>>> About `StateStoreMigratedException`:
> >>>>>
> >>>>> Why is it only thrown if the state is REBALANCING? A store might be
> >>>>> migrated during a rebalance, and Kafka Streams might resume back to
> >>>>> RUNNING state and afterward somebody tries to use an old store
> handle.
> >>>>> Also, if state is REBALANCING, should we throw
> >>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
> >> state.
> >>>>>
> >>>>
> >>>> Thank you point this, already updated.
> >>>>
> >>>>
> >>>> Why do we need to distinguish between
> `KafkaStreamsNotRunningException`
> >>>>> and `StateStoreNotAvailableException`?
> >>>>>
> >>>>
> >>>> `KafkaStreamsNotRunningException` may be caused by various reasons, I
> >> think
> >>>> it would be helpful that the
> >>>> user can distinguish whether it is caused by the state store closed.
> >>>> (Maybe I am wrong...)
> >>>>
> >>>>
> >>>> Last, why do we distinguish between `KafkaStreams` instance and
> >>>>> `StreamsThread`? To me, it seems we should always refer to the
> >> instance,
> >>>>> because that is the level of granularity in which we enable/disable
> >> IQ atm.
> >>>>>
> >>>>
> >>>> Totally agree. Do you mean the naming of state store exceptions?
> >>>> I don't have special reason to distinguish these two.
> >>>> Your suggestion look more reasonable for the exception naming.
> >>>>
> >>>>
> >>>> Last, for `StateStoreMigratedException`, I would add that a user need
> >> to
> >>>>> rediscover the store and cannot blindly retry as the store handle is
> >>>>> invalid and a new store handle must be retrieved. That is a
> difference
> >>>>> to `StreamThreadRebalancingException` that allows for "blind" retries
> >>>>> that either resolve (if the store is still on the same instance after
> >>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
> >> the
> >>>>> store was migrated away during rebalancing).
> >>>>>
> >>>>
> >>>> Nice, it's great! Thank you.
> >>>>
> >>>>
> >>>> The KIP already updated, please take a look. :)
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <
> matthias@confluent.io
> >>>
> >>>> wrote:
> >>>>
> >>>>> Any update on this KIP?
> >>>>>
> >>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >>>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> >>>>>>
> >>>>>> About `StateStoreMigratedException`:
> >>>>>>
> >>>>>> Why is it only thrown if the state is REBALANCING? A store might be
> >>>>>> migrated during a rebalance, and Kafka Streams might resume back to
> >>>>>> RUNNING state and afterward somebody tries to use an old store
> >> handle.
> >>>>>> Also, if state is REBALANCING, should we throw
> >>>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
> >>>>> state.
> >>>>>>
> >>>>>>
> >>>>>> Why do we need to distinguish between
> >> `KafkaStreamsNotRunningException`
> >>>>>> and `StateStoreNotAvailableException`?
> >>>>>>
> >>>>>>
> >>>>>> Last, why do we distinguish between `KafkaStreams` instance and
> >>>>>> `StreamsThread`? To me, it seems we should always refer to the
> >> instance,
> >>>>>> because that is the level of granularity in which we enable/disable
> >> IQ
> >>>>> atm.
> >>>>>>
> >>>>>>
> >>>>>> Last, for `StateStoreMigratedException`, I would add that a user
> >> need to
> >>>>>> rediscover the store and cannot blindly retry as the store handle is
> >>>>>> invalid and a new store handle must be retrieved. That is a
> >> difference
> >>>>>> to `StreamThreadRebalancingException` that allows for "blind"
> retries
> >>>>>> that either resolve (if the store is still on the same instance
> after
> >>>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
> >> the
> >>>>>> store was migrated away during rebalancing).
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >> <https://shorturl.at/CDNT9>
> >>>>> <https://shorturl.at/CDNT9>
> >>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>>
> >>>>>>> Please use the following instead: https://shorturl.at/bkKQU
> >>>>>>>
> >>>>>>>
> >>>>>>> ---
> >>>>>>> Vito
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw>
> >> wrote:
> >>>>>>>
> >>>>>>>> Thanks, Matthias!
> >>>>>>>>
> >>>>>>>>> About `StreamThreadNotStartedException`:
> >>>>>>>>
> >>>>>>>> Thank you for explanation. I agree with your opinion.
> >>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>>
> >>>>>>>> For the case that corresponding thread crashes after we handed out
> >> the
> >>>>>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
> >>>>>>>> `StateStoreMigratedException`.
> >>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>>> `KafkaStreamsNotRunningException` when stream thread is not
> >> running(
> >>>>>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException`
> >> when
> >>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not
> >> need
> >>>>> to
> >>>>>>>> add a new type for this case. Does that make sense?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> About `KafkaStreamsNotRunningException` vs
> >>>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>>
> >>>>>>>> I understand your point. I rename
> >> `StreamThreadNotRunningException` to
> >>>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> About check unknown state store names:
> >>>>>>>> Thank you for the hint. I add a new type
> >> `UnknownStateStoreException`
> >>>>> for
> >>>>>>>> this case.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Also, we should still have fatal exception
> >>>>>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
> >>>>>>>>
> >>>>>>>> Thank you point this, already add it again.
> >>>>>>>>
> >>>>>>>> The KIP already updated, please take a look.
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Vito
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Vito,
It's still unclear to me what the advantage is, to have both
`StreamsNotRunningException` and `StateStoreNotAvailableException`?
For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
and thus, for a user point of view, why does it matter if the store is
closed on not? I don't understand why/how this information would be
useful? Do you have a concrete example in mind how a user would react
differently to both exceptions?
@Vinoth: about `StreamsRebalancingException` -- to me, it seems best to
actually do this on a per-query basis, ie, have an overload
`KafkaStreams#store(...)` that takes a boolean flag that allow to
_disable_ the exception and opt-in to query a active store during
recovery. However, as KIP-535 actually introduces this change in
behavior, I think KIP-216 should not cover this, but KIP-535 should be
updated. I'll follow up on the other KIP thread to raise this point.
-Matthias
On 1/11/20 12:26 AM, Vito Jeng wrote:
> Hi, Matthias & Vinoth,
>
> Thanks for the feedback.
>
>> What is still unclear to me is, what we gain by having both
>> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
>> state store is closed on not -- I can't query it anyway? Maybe I miss
>> something thought?
>
> Yes, both `StreamsNotRunningException` and
> `StateStoreNotAvailableException` are fatal exception.
> But `StateStoreNotAvailableException` is fatal exception about state store
> related.
> I think it would be helpful that if user need to distinguish these two
> different case to handle it.
>
> I'm not very sure, does that make sense?
>
>
> ---
> Vito
>
>
> On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <vi...@apache.org> wrote:
>
>> +1 on merging `StreamsNotRunningException` and
>> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
>> its best to have these exceptions be about the state store (and not streams
>> state), to easier understanding.
>>
>> Additionally, KIP-535 allows for querying of state stores in rebalancing
>> state. So do we need the StreamsRebalancingException?
>>
>>
>> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io> wrote:
>>> Sorry that I dropped the ball on this...
>>>
>>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
>>> thread.
>>>
>>> What is still unclear to me is, what we gain by having both
>>> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
>>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
>>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
>>> state store is closed on not -- I can't query it anyway? Maybe I miss
>>> something thought?
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>>>> Sorry for the late reply, thanks for the review.
>>>>
>>>>
>>>>> About `StateStoreMigratedException`:
>>>>>
>>>>> Why is it only thrown if the state is REBALANCING? A store might be
>>>>> migrated during a rebalance, and Kafka Streams might resume back to
>>>>> RUNNING state and afterward somebody tries to use an old store handle.
>>>>> Also, if state is REBALANCING, should we throw
>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
>> state.
>>>>>
>>>>
>>>> Thank you point this, already updated.
>>>>
>>>>
>>>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
>>>>> and `StateStoreNotAvailableException`?
>>>>>
>>>>
>>>> `KafkaStreamsNotRunningException` may be caused by various reasons, I
>> think
>>>> it would be helpful that the
>>>> user can distinguish whether it is caused by the state store closed.
>>>> (Maybe I am wrong...)
>>>>
>>>>
>>>> Last, why do we distinguish between `KafkaStreams` instance and
>>>>> `StreamsThread`? To me, it seems we should always refer to the
>> instance,
>>>>> because that is the level of granularity in which we enable/disable
>> IQ atm.
>>>>>
>>>>
>>>> Totally agree. Do you mean the naming of state store exceptions?
>>>> I don't have special reason to distinguish these two.
>>>> Your suggestion look more reasonable for the exception naming.
>>>>
>>>>
>>>> Last, for `StateStoreMigratedException`, I would add that a user need
>> to
>>>>> rediscover the store and cannot blindly retry as the store handle is
>>>>> invalid and a new store handle must be retrieved. That is a difference
>>>>> to `StreamThreadRebalancingException` that allows for "blind" retries
>>>>> that either resolve (if the store is still on the same instance after
>>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
>> the
>>>>> store was migrated away during rebalancing).
>>>>>
>>>>
>>>> Nice, it's great! Thank you.
>>>>
>>>>
>>>> The KIP already updated, please take a look. :)
>>>>
>>>>
>>>>
>>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <matthias@confluent.io
>>>
>>>> wrote:
>>>>
>>>>> Any update on this KIP?
>>>>>
>>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>>>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
>>>>>>
>>>>>> About `StateStoreMigratedException`:
>>>>>>
>>>>>> Why is it only thrown if the state is REBALANCING? A store might be
>>>>>> migrated during a rebalance, and Kafka Streams might resume back to
>>>>>> RUNNING state and afterward somebody tries to use an old store
>> handle.
>>>>>> Also, if state is REBALANCING, should we throw
>>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
>>>>> state.
>>>>>>
>>>>>>
>>>>>> Why do we need to distinguish between
>> `KafkaStreamsNotRunningException`
>>>>>> and `StateStoreNotAvailableException`?
>>>>>>
>>>>>>
>>>>>> Last, why do we distinguish between `KafkaStreams` instance and
>>>>>> `StreamsThread`? To me, it seems we should always refer to the
>> instance,
>>>>>> because that is the level of granularity in which we enable/disable
>> IQ
>>>>> atm.
>>>>>>
>>>>>>
>>>>>> Last, for `StateStoreMigratedException`, I would add that a user
>> need to
>>>>>> rediscover the store and cannot blindly retry as the store handle is
>>>>>> invalid and a new store handle must be retrieved. That is a
>> difference
>>>>>> to `StreamThreadRebalancingException` that allows for "blind" retries
>>>>>> that either resolve (if the store is still on the same instance after
>>>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
>> the
>>>>>> store was migrated away during rebalancing).
>>>>>>
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9>
>>>>> <https://shorturl.at/CDNT9>
>>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>>>>>>>
>>>>>>> Please use the following instead: https://shorturl.at/bkKQU
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Vito
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw>
>> wrote:
>>>>>>>
>>>>>>>> Thanks, Matthias!
>>>>>>>>
>>>>>>>>> About `StreamThreadNotStartedException`:
>>>>>>>>
>>>>>>>> Thank you for explanation. I agree with your opinion.
>>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>>
>>>>>>>> For the case that corresponding thread crashes after we handed out
>> the
>>>>>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
>>>>>>>> `StateStoreMigratedException`.
>>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>>>>>>>> `KafkaStreamsNotRunningException` when stream thread is not
>> running(
>>>>>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException`
>> when
>>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not
>> need
>>>>> to
>>>>>>>> add a new type for this case. Does that make sense?
>>>>>>>>
>>>>>>>>
>>>>>>>>> About `KafkaStreamsNotRunningException` vs
>>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>>
>>>>>>>> I understand your point. I rename
>> `StreamThreadNotRunningException` to
>>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>>
>>>>>>>>
>>>>>>>> About check unknown state store names:
>>>>>>>> Thank you for the hint. I add a new type
>> `UnknownStateStoreException`
>>>>> for
>>>>>>>> this case.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Also, we should still have fatal exception
>>>>>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
>>>>>>>>
>>>>>>>> Thank you point this, already add it again.
>>>>>>>>
>>>>>>>> The KIP already updated, please take a look.
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vinoth Chandar <vc...@confluent.io>.
+1 on a streams config to control this behavior. I have no strong opinions
on the default, but I would pick allowing to query if standbys are enabled
else throw the exception..
But we can keep it simpler, throw exception by default and have a flag to
turn it off, as you suggest as well.
On Thu, Jan 9, 2020 at 12:01 PM Matthias J. Sax <ma...@confluent.io>
wrote:
> Good question about `StreamsRebalancingException` -- when this KIP was
> started, KIP-535 was not on the horizon yet.
>
> What I am wondering is, if we should allow people to opt-in into
> querying during a rebalance, or to be more precise during a restore (if
> a state store is not migrated, it will be up-to-date during a rebalance
> and can be queried returning correct, ie, non-stall, data)?
>
> Otherwise, if people want to get only correct results, ie, they never
> want to query stall state, they have no way to implement it, because
> they are always subject to a race condition.
>
> For this case, we could have a `StateStoreIsRecoveringException` (or
> similar) that is only throw during a restore phases (and people can
> opt-in / opt-out if this exception should be throws or not, ie, if they
> want to query stall state during recovery or not).
>
> It's unclear to me though atm, how a user would opt-in/opt-out and what
> the default should be (maybe better to throw the exception by default to
> have strong consistency guarantees by default?)
>
>
> -Matthias
>
>
> On 1/9/20 11:35 AM, Vinoth Chandar wrote:
> > +1 on merging `StreamsNotRunningException` and
> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
> its best to have these exceptions be about the state store (and not streams
> state), to easier understanding.
> >
> > Additionally, KIP-535 allows for querying of state stores in rebalancing
> state. So do we need the StreamsRebalancingException?
> >
> >
> > On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io>
> wrote:
> >> Sorry that I dropped the ball on this...
> >>
> >> Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
> >> thread.
> >>
> >> What is still unclear to me is, what we gain by having both
> >> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> >> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> >> state store is closed on not -- I can't query it anyway? Maybe I miss
> >> something thought?
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 11/3/19 6:07 PM, Vito Jeng wrote:
> >>> Sorry for the late reply, thanks for the review.
> >>>
> >>>
> >>>> About `StateStoreMigratedException`:
> >>>>
> >>>> Why is it only thrown if the state is REBALANCING? A store might be
> >>>> migrated during a rebalance, and Kafka Streams might resume back to
> >>>> RUNNING state and afterward somebody tries to use an old store handle.
> >>>> Also, if state is REBALANCING, should we throw
> >>>> `StreamThreadRebalancingException`? Hence, I think
> >>>> `StateStoreMigratedException` does only make sense during `RUNNING`
> state.
> >>>>
> >>>
> >>> Thank you point this, already updated.
> >>>
> >>>
> >>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
> >>>> and `StateStoreNotAvailableException`?
> >>>>
> >>>
> >>> `KafkaStreamsNotRunningException` may be caused by various reasons, I
> think
> >>> it would be helpful that the
> >>> user can distinguish whether it is caused by the state store closed.
> >>> (Maybe I am wrong...)
> >>>
> >>>
> >>> Last, why do we distinguish between `KafkaStreams` instance and
> >>>> `StreamsThread`? To me, it seems we should always refer to the
> instance,
> >>>> because that is the level of granularity in which we enable/disable
> IQ atm.
> >>>>
> >>>
> >>> Totally agree. Do you mean the naming of state store exceptions?
> >>> I don't have special reason to distinguish these two.
> >>> Your suggestion look more reasonable for the exception naming.
> >>>
> >>>
> >>> Last, for `StateStoreMigratedException`, I would add that a user need
> to
> >>>> rediscover the store and cannot blindly retry as the store handle is
> >>>> invalid and a new store handle must be retrieved. That is a difference
> >>>> to `StreamThreadRebalancingException` that allows for "blind" retries
> >>>> that either resolve (if the store is still on the same instance after
> >>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
> the
> >>>> store was migrated away during rebalancing).
> >>>>
> >>>
> >>> Nice, it's great! Thank you.
> >>>
> >>>
> >>> The KIP already updated, please take a look. :)
> >>>
> >>>
> >>>
> >>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <matthias@confluent.io
> >
> >>> wrote:
> >>>
> >>>> Any update on this KIP?
> >>>>
> >>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> >>>>>
> >>>>> About `StateStoreMigratedException`:
> >>>>>
> >>>>> Why is it only thrown if the state is REBALANCING? A store might be
> >>>>> migrated during a rebalance, and Kafka Streams might resume back to
> >>>>> RUNNING state and afterward somebody tries to use an old store
> handle.
> >>>>> Also, if state is REBALANCING, should we throw
> >>>>> `StreamThreadRebalancingException`? Hence, I think
> >>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
> >>>> state.
> >>>>>
> >>>>>
> >>>>> Why do we need to distinguish between
> `KafkaStreamsNotRunningException`
> >>>>> and `StateStoreNotAvailableException`?
> >>>>>
> >>>>>
> >>>>> Last, why do we distinguish between `KafkaStreams` instance and
> >>>>> `StreamsThread`? To me, it seems we should always refer to the
> instance,
> >>>>> because that is the level of granularity in which we enable/disable
> IQ
> >>>> atm.
> >>>>>
> >>>>>
> >>>>> Last, for `StateStoreMigratedException`, I would add that a user
> need to
> >>>>> rediscover the store and cannot blindly retry as the store handle is
> >>>>> invalid and a new store handle must be retrieved. That is a
> difference
> >>>>> to `StreamThreadRebalancingException` that allows for "blind" retries
> >>>>> that either resolve (if the store is still on the same instance after
> >>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if
> the
> >>>>> store was migrated away during rebalancing).
> >>>>>
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >>>>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >>>> <https://shorturl.at/CDNT9>
> >>>>>> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>>>
> >>>>>> Please use the following instead: https://shorturl.at/bkKQU
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw>
> wrote:
> >>>>>>
> >>>>>>> Thanks, Matthias!
> >>>>>>>
> >>>>>>>> About `StreamThreadNotStartedException`:
> >>>>>>>
> >>>>>>> Thank you for explanation. I agree with your opinion.
> >>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>>>>>> `StreamThreadNotStartedException`.
> >>>>>>>
> >>>>>>> For the case that corresponding thread crashes after we handed out
> the
> >>>>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
> >>>>>>> `StateStoreMigratedException`.
> >>>>>>> In `StreamThreadStateStoreProvider`, we would throw
> >>>>>>> `KafkaStreamsNotRunningException` when stream thread is not
> running(
> >>>>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException`
> when
> >>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not
> need
> >>>> to
> >>>>>>> add a new type for this case. Does that make sense?
> >>>>>>>
> >>>>>>>
> >>>>>>>> About `KafkaStreamsNotRunningException` vs
> >>>>>>> `StreamThreadNotRunningException`:
> >>>>>>>
> >>>>>>> I understand your point. I rename
> `StreamThreadNotRunningException` to
> >>>>>>> `KafkaStreamsNotRunningException`.
> >>>>>>>
> >>>>>>>
> >>>>>>> About check unknown state store names:
> >>>>>>> Thank you for the hint. I add a new type
> `UnknownStateStoreException`
> >>>> for
> >>>>>>> this case.
> >>>>>>>
> >>>>>>>
> >>>>>>>> Also, we should still have fatal exception
> >>>>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
> >>>>>>>
> >>>>>>> Thank you point this, already add it again.
> >>>>>>>
> >>>>>>> The KIP already updated, please take a look.
> >>>>>>>
> >>>>>>> ---
> >>>>>>> Vito
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Good question about `StreamsRebalancingException` -- when this KIP was
started, KIP-535 was not on the horizon yet.
What I am wondering is, if we should allow people to opt-in into
querying during a rebalance, or to be more precise during a restore (if
a state store is not migrated, it will be up-to-date during a rebalance
and can be queried returning correct, ie, non-stall, data)?
Otherwise, if people want to get only correct results, ie, they never
want to query stall state, they have no way to implement it, because
they are always subject to a race condition.
For this case, we could have a `StateStoreIsRecoveringException` (or
similar) that is only throw during a restore phases (and people can
opt-in / opt-out if this exception should be throws or not, ie, if they
want to query stall state during recovery or not).
It's unclear to me though atm, how a user would opt-in/opt-out and what
the default should be (maybe better to throw the exception by default to
have strong consistency guarantees by default?)
-Matthias
On 1/9/20 11:35 AM, Vinoth Chandar wrote:
> +1 on merging `StreamsNotRunningException` and `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO its best to have these exceptions be about the state store (and not streams state), to easier understanding.
>
> Additionally, KIP-535 allows for querying of state stores in rebalancing state. So do we need the StreamsRebalancingException?
>
>
> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io> wrote:
>> Sorry that I dropped the ball on this...
>>
>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
>> thread.
>>
>> What is still unclear to me is, what we gain by having both
>> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
>> state store is closed on not -- I can't query it anyway? Maybe I miss
>> something thought?
>>
>>
>> -Matthias
>>
>>
>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>>> Sorry for the late reply, thanks for the review.
>>>
>>>
>>>> About `StateStoreMigratedException`:
>>>>
>>>> Why is it only thrown if the state is REBALANCING? A store might be
>>>> migrated during a rebalance, and Kafka Streams might resume back to
>>>> RUNNING state and afterward somebody tries to use an old store handle.
>>>> Also, if state is REBALANCING, should we throw
>>>> `StreamThreadRebalancingException`? Hence, I think
>>>> `StateStoreMigratedException` does only make sense during `RUNNING` state.
>>>>
>>>
>>> Thank you point this, already updated.
>>>
>>>
>>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
>>>> and `StateStoreNotAvailableException`?
>>>>
>>>
>>> `KafkaStreamsNotRunningException` may be caused by various reasons, I think
>>> it would be helpful that the
>>> user can distinguish whether it is caused by the state store closed.
>>> (Maybe I am wrong...)
>>>
>>>
>>> Last, why do we distinguish between `KafkaStreams` instance and
>>>> `StreamsThread`? To me, it seems we should always refer to the instance,
>>>> because that is the level of granularity in which we enable/disable IQ atm.
>>>>
>>>
>>> Totally agree. Do you mean the naming of state store exceptions?
>>> I don't have special reason to distinguish these two.
>>> Your suggestion look more reasonable for the exception naming.
>>>
>>>
>>> Last, for `StateStoreMigratedException`, I would add that a user need to
>>>> rediscover the store and cannot blindly retry as the store handle is
>>>> invalid and a new store handle must be retrieved. That is a difference
>>>> to `StreamThreadRebalancingException` that allows for "blind" retries
>>>> that either resolve (if the store is still on the same instance after
>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if the
>>>> store was migrated away during rebalancing).
>>>>
>>>
>>> Nice, it's great! Thank you.
>>>
>>>
>>> The KIP already updated, please take a look. :)
>>>
>>>
>>>
>>> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Any update on this KIP?
>>>>
>>>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>>>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
>>>>>
>>>>> About `StateStoreMigratedException`:
>>>>>
>>>>> Why is it only thrown if the state is REBALANCING? A store might be
>>>>> migrated during a rebalance, and Kafka Streams might resume back to
>>>>> RUNNING state and afterward somebody tries to use an old store handle.
>>>>> Also, if state is REBALANCING, should we throw
>>>>> `StreamThreadRebalancingException`? Hence, I think
>>>>> `StateStoreMigratedException` does only make sense during `RUNNING`
>>>> state.
>>>>>
>>>>>
>>>>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
>>>>> and `StateStoreNotAvailableException`?
>>>>>
>>>>>
>>>>> Last, why do we distinguish between `KafkaStreams` instance and
>>>>> `StreamsThread`? To me, it seems we should always refer to the instance,
>>>>> because that is the level of granularity in which we enable/disable IQ
>>>> atm.
>>>>>
>>>>>
>>>>> Last, for `StateStoreMigratedException`, I would add that a user need to
>>>>> rediscover the store and cannot blindly retry as the store handle is
>>>>> invalid and a new store handle must be retrieved. That is a difference
>>>>> to `StreamThreadRebalancingException` that allows for "blind" retries
>>>>> that either resolve (if the store is still on the same instance after
>>>>> rebalancing finishes, or changes to `StateStoreMigratedException` if the
>>>>> store was migrated away during rebalancing).
>>>>>
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>>>>>> My bad. The short link `https://shorturl.at/CDNT9`
>>>> <https://shorturl.at/CDNT9>
>>>>>> <https://shorturl.at/CDNT9> seems incorrect.
>>>>>>
>>>>>> Please use the following instead: https://shorturl.at/bkKQU
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Vito
>>>>>>
>>>>>>
>>>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>>>>>>
>>>>>>> Thanks, Matthias!
>>>>>>>
>>>>>>>> About `StreamThreadNotStartedException`:
>>>>>>>
>>>>>>> Thank you for explanation. I agree with your opinion.
>>>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>>>>>>> `StreamThreadNotStartedException`.
>>>>>>>
>>>>>>> For the case that corresponding thread crashes after we handed out the
>>>>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
>>>>>>> `StateStoreMigratedException`.
>>>>>>> In `StreamThreadStateStoreProvider`, we would throw
>>>>>>> `KafkaStreamsNotRunningException` when stream thread is not running(
>>>>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
>>>>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not need
>>>> to
>>>>>>> add a new type for this case. Does that make sense?
>>>>>>>
>>>>>>>
>>>>>>>> About `KafkaStreamsNotRunningException` vs
>>>>>>> `StreamThreadNotRunningException`:
>>>>>>>
>>>>>>> I understand your point. I rename `StreamThreadNotRunningException` to
>>>>>>> `KafkaStreamsNotRunningException`.
>>>>>>>
>>>>>>>
>>>>>>> About check unknown state store names:
>>>>>>> Thank you for the hint. I add a new type `UnknownStateStoreException`
>>>> for
>>>>>>> this case.
>>>>>>>
>>>>>>>
>>>>>>>> Also, we should still have fatal exception
>>>>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
>>>>>>>
>>>>>>> Thank you point this, already add it again.
>>>>>>>
>>>>>>> The KIP already updated, please take a look.
>>>>>>>
>>>>>>> ---
>>>>>>> Vito
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, Matthias & Vinoth,
Thanks for the feedback.
> What is still unclear to me is, what we gain by having both
> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> state store is closed on not -- I can't query it anyway? Maybe I miss
> something thought?
Yes, both `StreamsNotRunningException` and
`StateStoreNotAvailableException` are fatal exception.
But `StateStoreNotAvailableException` is fatal exception about state store
related.
I think it would be helpful that if user need to distinguish these two
different case to handle it.
I'm not very sure, does that make sense?
---
Vito
On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar <vi...@apache.org> wrote:
> +1 on merging `StreamsNotRunningException` and
> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
> its best to have these exceptions be about the state store (and not streams
> state), to easier understanding.
>
> Additionally, KIP-535 allows for querying of state stores in rebalancing
> state. So do we need the StreamsRebalancingException?
>
>
> On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io> wrote:
> > Sorry that I dropped the ball on this...
> >
> > Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
> > thread.
> >
> > What is still unclear to me is, what we gain by having both
> > `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> > exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> > NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> > state store is closed on not -- I can't query it anyway? Maybe I miss
> > something thought?
> >
> >
> > -Matthias
> >
> >
> > On 11/3/19 6:07 PM, Vito Jeng wrote:
> > > Sorry for the late reply, thanks for the review.
> > >
> > >
> > >> About `StateStoreMigratedException`:
> > >>
> > >> Why is it only thrown if the state is REBALANCING? A store might be
> > >> migrated during a rebalance, and Kafka Streams might resume back to
> > >> RUNNING state and afterward somebody tries to use an old store handle.
> > >> Also, if state is REBALANCING, should we throw
> > >> `StreamThreadRebalancingException`? Hence, I think
> > >> `StateStoreMigratedException` does only make sense during `RUNNING`
> state.
> > >>
> > >
> > > Thank you point this, already updated.
> > >
> > >
> > > Why do we need to distinguish between `KafkaStreamsNotRunningException`
> > >> and `StateStoreNotAvailableException`?
> > >>
> > >
> > > `KafkaStreamsNotRunningException` may be caused by various reasons, I
> think
> > > it would be helpful that the
> > > user can distinguish whether it is caused by the state store closed.
> > > (Maybe I am wrong...)
> > >
> > >
> > > Last, why do we distinguish between `KafkaStreams` instance and
> > >> `StreamsThread`? To me, it seems we should always refer to the
> instance,
> > >> because that is the level of granularity in which we enable/disable
> IQ atm.
> > >>
> > >
> > > Totally agree. Do you mean the naming of state store exceptions?
> > > I don't have special reason to distinguish these two.
> > > Your suggestion look more reasonable for the exception naming.
> > >
> > >
> > > Last, for `StateStoreMigratedException`, I would add that a user need
> to
> > >> rediscover the store and cannot blindly retry as the store handle is
> > >> invalid and a new store handle must be retrieved. That is a difference
> > >> to `StreamThreadRebalancingException` that allows for "blind" retries
> > >> that either resolve (if the store is still on the same instance after
> > >> rebalancing finishes, or changes to `StateStoreMigratedException` if
> the
> > >> store was migrated away during rebalancing).
> > >>
> > >
> > > Nice, it's great! Thank you.
> > >
> > >
> > > The KIP already updated, please take a look. :)
> > >
> > >
> > >
> > > On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <matthias@confluent.io
> >
> > > wrote:
> > >
> > >> Any update on this KIP?
> > >>
> > >> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > >>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> > >>>
> > >>> About `StateStoreMigratedException`:
> > >>>
> > >>> Why is it only thrown if the state is REBALANCING? A store might be
> > >>> migrated during a rebalance, and Kafka Streams might resume back to
> > >>> RUNNING state and afterward somebody tries to use an old store
> handle.
> > >>> Also, if state is REBALANCING, should we throw
> > >>> `StreamThreadRebalancingException`? Hence, I think
> > >>> `StateStoreMigratedException` does only make sense during `RUNNING`
> > >> state.
> > >>>
> > >>>
> > >>> Why do we need to distinguish between
> `KafkaStreamsNotRunningException`
> > >>> and `StateStoreNotAvailableException`?
> > >>>
> > >>>
> > >>> Last, why do we distinguish between `KafkaStreams` instance and
> > >>> `StreamsThread`? To me, it seems we should always refer to the
> instance,
> > >>> because that is the level of granularity in which we enable/disable
> IQ
> > >> atm.
> > >>>
> > >>>
> > >>> Last, for `StateStoreMigratedException`, I would add that a user
> need to
> > >>> rediscover the store and cannot blindly retry as the store handle is
> > >>> invalid and a new store handle must be retrieved. That is a
> difference
> > >>> to `StreamThreadRebalancingException` that allows for "blind" retries
> > >>> that either resolve (if the store is still on the same instance after
> > >>> rebalancing finishes, or changes to `StateStoreMigratedException` if
> the
> > >>> store was migrated away during rebalancing).
> > >>>
> > >>>
> > >>>
> > >>> -Matthias
> > >>>
> > >>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> > >>>> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> > >> <https://shorturl.at/CDNT9>
> > >>>> <https://shorturl.at/CDNT9> seems incorrect.
> > >>>>
> > >>>> Please use the following instead: https://shorturl.at/bkKQU
> > >>>>
> > >>>>
> > >>>> ---
> > >>>> Vito
> > >>>>
> > >>>>
> > >>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw>
> wrote:
> > >>>>
> > >>>>> Thanks, Matthias!
> > >>>>>
> > >>>>>> About `StreamThreadNotStartedException`:
> > >>>>>
> > >>>>> Thank you for explanation. I agree with your opinion.
> > >>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> > >>>>> `StreamThreadNotStartedException`.
> > >>>>>
> > >>>>> For the case that corresponding thread crashes after we handed out
> the
> > >>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
> > >>>>> `StateStoreMigratedException`.
> > >>>>> In `StreamThreadStateStoreProvider`, we would throw
> > >>>>> `KafkaStreamsNotRunningException` when stream thread is not
> running(
> > >>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException`
> when
> > >>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not
> need
> > >> to
> > >>>>> add a new type for this case. Does that make sense?
> > >>>>>
> > >>>>>
> > >>>>>> About `KafkaStreamsNotRunningException` vs
> > >>>>> `StreamThreadNotRunningException`:
> > >>>>>
> > >>>>> I understand your point. I rename
> `StreamThreadNotRunningException` to
> > >>>>> `KafkaStreamsNotRunningException`.
> > >>>>>
> > >>>>>
> > >>>>> About check unknown state store names:
> > >>>>> Thank you for the hint. I add a new type
> `UnknownStateStoreException`
> > >> for
> > >>>>> this case.
> > >>>>>
> > >>>>>
> > >>>>>> Also, we should still have fatal exception
> > >>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
> > >>>>>
> > >>>>> Thank you point this, already add it again.
> > >>>>>
> > >>>>> The KIP already updated, please take a look.
> > >>>>>
> > >>>>> ---
> > >>>>> Vito
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >
> >
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors
Posted by Vinoth Chandar <vi...@apache.org>.
+1 on merging `StreamsNotRunningException` and `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO its best to have these exceptions be about the state store (and not streams state), to easier understanding.
Additionally, KIP-535 allows for querying of state stores in rebalancing state. So do we need the StreamsRebalancingException?
On 2020/01/09 03:38:11, "Matthias J. Sax" <ma...@confluent.io> wrote:
> Sorry that I dropped the ball on this...
>
> Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
> thread.
>
> What is still unclear to me is, what we gain by having both
> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> state store is closed on not -- I can't query it anyway? Maybe I miss
> something thought?
>
>
> -Matthias
>
>
> On 11/3/19 6:07 PM, Vito Jeng wrote:
> > Sorry for the late reply, thanks for the review.
> >
> >
> >> About `StateStoreMigratedException`:
> >>
> >> Why is it only thrown if the state is REBALANCING? A store might be
> >> migrated during a rebalance, and Kafka Streams might resume back to
> >> RUNNING state and afterward somebody tries to use an old store handle.
> >> Also, if state is REBALANCING, should we throw
> >> `StreamThreadRebalancingException`? Hence, I think
> >> `StateStoreMigratedException` does only make sense during `RUNNING` state.
> >>
> >
> > Thank you point this, already updated.
> >
> >
> > Why do we need to distinguish between `KafkaStreamsNotRunningException`
> >> and `StateStoreNotAvailableException`?
> >>
> >
> > `KafkaStreamsNotRunningException` may be caused by various reasons, I think
> > it would be helpful that the
> > user can distinguish whether it is caused by the state store closed.
> > (Maybe I am wrong...)
> >
> >
> > Last, why do we distinguish between `KafkaStreams` instance and
> >> `StreamsThread`? To me, it seems we should always refer to the instance,
> >> because that is the level of granularity in which we enable/disable IQ atm.
> >>
> >
> > Totally agree. Do you mean the naming of state store exceptions?
> > I don't have special reason to distinguish these two.
> > Your suggestion look more reasonable for the exception naming.
> >
> >
> > Last, for `StateStoreMigratedException`, I would add that a user need to
> >> rediscover the store and cannot blindly retry as the store handle is
> >> invalid and a new store handle must be retrieved. That is a difference
> >> to `StreamThreadRebalancingException` that allows for "blind" retries
> >> that either resolve (if the store is still on the same instance after
> >> rebalancing finishes, or changes to `StateStoreMigratedException` if the
> >> store was migrated away during rebalancing).
> >>
> >
> > Nice, it's great! Thank you.
> >
> >
> > The KIP already updated, please take a look. :)
> >
> >
> >
> > On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Any update on this KIP?
> >>
> >> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> >>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
> >>>
> >>> About `StateStoreMigratedException`:
> >>>
> >>> Why is it only thrown if the state is REBALANCING? A store might be
> >>> migrated during a rebalance, and Kafka Streams might resume back to
> >>> RUNNING state and afterward somebody tries to use an old store handle.
> >>> Also, if state is REBALANCING, should we throw
> >>> `StreamThreadRebalancingException`? Hence, I think
> >>> `StateStoreMigratedException` does only make sense during `RUNNING`
> >> state.
> >>>
> >>>
> >>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
> >>> and `StateStoreNotAvailableException`?
> >>>
> >>>
> >>> Last, why do we distinguish between `KafkaStreams` instance and
> >>> `StreamsThread`? To me, it seems we should always refer to the instance,
> >>> because that is the level of granularity in which we enable/disable IQ
> >> atm.
> >>>
> >>>
> >>> Last, for `StateStoreMigratedException`, I would add that a user need to
> >>> rediscover the store and cannot blindly retry as the store handle is
> >>> invalid and a new store handle must be retrieved. That is a difference
> >>> to `StreamThreadRebalancingException` that allows for "blind" retries
> >>> that either resolve (if the store is still on the same instance after
> >>> rebalancing finishes, or changes to `StateStoreMigratedException` if the
> >>> store was migrated away during rebalancing).
> >>>
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 8/9/19 10:20 AM, Vito Jeng wrote:
> >>>> My bad. The short link `https://shorturl.at/CDNT9`
> >> <https://shorturl.at/CDNT9>
> >>>> <https://shorturl.at/CDNT9> seems incorrect.
> >>>>
> >>>> Please use the following instead: https://shorturl.at/bkKQU
> >>>>
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>>
> >>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> >>>>
> >>>>> Thanks, Matthias!
> >>>>>
> >>>>>> About `StreamThreadNotStartedException`:
> >>>>>
> >>>>> Thank you for explanation. I agree with your opinion.
> >>>>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>>>> `StreamThreadNotStartedException`.
> >>>>>
> >>>>> For the case that corresponding thread crashes after we handed out the
> >>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
> >>>>> `StateStoreMigratedException`.
> >>>>> In `StreamThreadStateStoreProvider`, we would throw
> >>>>> `KafkaStreamsNotRunningException` when stream thread is not running(
> >>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
> >>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not need
> >> to
> >>>>> add a new type for this case. Does that make sense?
> >>>>>
> >>>>>
> >>>>>> About `KafkaStreamsNotRunningException` vs
> >>>>> `StreamThreadNotRunningException`:
> >>>>>
> >>>>> I understand your point. I rename `StreamThreadNotRunningException` to
> >>>>> `KafkaStreamsNotRunningException`.
> >>>>>
> >>>>>
> >>>>> About check unknown state store names:
> >>>>> Thank you for the hint. I add a new type `UnknownStateStoreException`
> >> for
> >>>>> this case.
> >>>>>
> >>>>>
> >>>>>> Also, we should still have fatal exception
> >>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
> >>>>>
> >>>>> Thank you point this, already add it again.
> >>>>>
> >>>>> The KIP already updated, please take a look.
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Sorry that I dropped the ball on this...
Thanks for updating the KIP. Overall LGTM now. Feel free to start a VOTE
thread.
What is still unclear to me is, what we gain by having both
`StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
state store is closed on not -- I can't query it anyway? Maybe I miss
something thought?
-Matthias
On 11/3/19 6:07 PM, Vito Jeng wrote:
> Sorry for the late reply, thanks for the review.
>
>
>> About `StateStoreMigratedException`:
>>
>> Why is it only thrown if the state is REBALANCING? A store might be
>> migrated during a rebalance, and Kafka Streams might resume back to
>> RUNNING state and afterward somebody tries to use an old store handle.
>> Also, if state is REBALANCING, should we throw
>> `StreamThreadRebalancingException`? Hence, I think
>> `StateStoreMigratedException` does only make sense during `RUNNING` state.
>>
>
> Thank you point this, already updated.
>
>
> Why do we need to distinguish between `KafkaStreamsNotRunningException`
>> and `StateStoreNotAvailableException`?
>>
>
> `KafkaStreamsNotRunningException` may be caused by various reasons, I think
> it would be helpful that the
> user can distinguish whether it is caused by the state store closed.
> (Maybe I am wrong...)
>
>
> Last, why do we distinguish between `KafkaStreams` instance and
>> `StreamsThread`? To me, it seems we should always refer to the instance,
>> because that is the level of granularity in which we enable/disable IQ atm.
>>
>
> Totally agree. Do you mean the naming of state store exceptions?
> I don't have special reason to distinguish these two.
> Your suggestion look more reasonable for the exception naming.
>
>
> Last, for `StateStoreMigratedException`, I would add that a user need to
>> rediscover the store and cannot blindly retry as the store handle is
>> invalid and a new store handle must be retrieved. That is a difference
>> to `StreamThreadRebalancingException` that allows for "blind" retries
>> that either resolve (if the store is still on the same instance after
>> rebalancing finishes, or changes to `StateStoreMigratedException` if the
>> store was migrated away during rebalancing).
>>
>
> Nice, it's great! Thank you.
>
>
> The KIP already updated, please take a look. :)
>
>
>
> On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Any update on this KIP?
>>
>> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
>>> Sorry for the late reply. The 2.4 deadline kept us quite busy.
>>>
>>> About `StateStoreMigratedException`:
>>>
>>> Why is it only thrown if the state is REBALANCING? A store might be
>>> migrated during a rebalance, and Kafka Streams might resume back to
>>> RUNNING state and afterward somebody tries to use an old store handle.
>>> Also, if state is REBALANCING, should we throw
>>> `StreamThreadRebalancingException`? Hence, I think
>>> `StateStoreMigratedException` does only make sense during `RUNNING`
>> state.
>>>
>>>
>>> Why do we need to distinguish between `KafkaStreamsNotRunningException`
>>> and `StateStoreNotAvailableException`?
>>>
>>>
>>> Last, why do we distinguish between `KafkaStreams` instance and
>>> `StreamsThread`? To me, it seems we should always refer to the instance,
>>> because that is the level of granularity in which we enable/disable IQ
>> atm.
>>>
>>>
>>> Last, for `StateStoreMigratedException`, I would add that a user need to
>>> rediscover the store and cannot blindly retry as the store handle is
>>> invalid and a new store handle must be retrieved. That is a difference
>>> to `StreamThreadRebalancingException` that allows for "blind" retries
>>> that either resolve (if the store is still on the same instance after
>>> rebalancing finishes, or changes to `StateStoreMigratedException` if the
>>> store was migrated away during rebalancing).
>>>
>>>
>>>
>>> -Matthias
>>>
>>> On 8/9/19 10:20 AM, Vito Jeng wrote:
>>>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9>
>>>> <https://shorturl.at/CDNT9> seems incorrect.
>>>>
>>>> Please use the following instead: https://shorturl.at/bkKQU
>>>>
>>>>
>>>> ---
>>>> Vito
>>>>
>>>>
>>>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>>>>
>>>>> Thanks, Matthias!
>>>>>
>>>>>> About `StreamThreadNotStartedException`:
>>>>>
>>>>> Thank you for explanation. I agree with your opinion.
>>>>> `CompositeReadOnlyXxxStore#get()` would never throw
>>>>> `StreamThreadNotStartedException`.
>>>>>
>>>>> For the case that corresponding thread crashes after we handed out the
>>>>> store handle. We may throw `KafkaStreamsNotRunningException` or
>>>>> `StateStoreMigratedException`.
>>>>> In `StreamThreadStateStoreProvider`, we would throw
>>>>> `KafkaStreamsNotRunningException` when stream thread is not running(
>>>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
>>>>> store is closed(https://shorturl.at/hrvAN). So I think we do not need
>> to
>>>>> add a new type for this case. Does that make sense?
>>>>>
>>>>>
>>>>>> About `KafkaStreamsNotRunningException` vs
>>>>> `StreamThreadNotRunningException`:
>>>>>
>>>>> I understand your point. I rename `StreamThreadNotRunningException` to
>>>>> `KafkaStreamsNotRunningException`.
>>>>>
>>>>>
>>>>> About check unknown state store names:
>>>>> Thank you for the hint. I add a new type `UnknownStateStoreException`
>> for
>>>>> this case.
>>>>>
>>>>>
>>>>>> Also, we should still have fatal exception
>>>>> `StateStoreNotAvailableException`? Not sure why you remove it?
>>>>>
>>>>> Thank you point this, already add it again.
>>>>>
>>>>> The KIP already updated, please take a look.
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Sorry for the late reply, thanks for the review.
> About `StateStoreMigratedException`:
>
> Why is it only thrown if the state is REBALANCING? A store might be
> migrated during a rebalance, and Kafka Streams might resume back to
> RUNNING state and afterward somebody tries to use an old store handle.
> Also, if state is REBALANCING, should we throw
> `StreamThreadRebalancingException`? Hence, I think
> `StateStoreMigratedException` does only make sense during `RUNNING` state.
>
Thank you point this, already updated.
Why do we need to distinguish between `KafkaStreamsNotRunningException`
> and `StateStoreNotAvailableException`?
>
`KafkaStreamsNotRunningException` may be caused by various reasons, I think
it would be helpful that the
user can distinguish whether it is caused by the state store closed.
(Maybe I am wrong...)
Last, why do we distinguish between `KafkaStreams` instance and
> `StreamsThread`? To me, it seems we should always refer to the instance,
> because that is the level of granularity in which we enable/disable IQ atm.
>
Totally agree. Do you mean the naming of state store exceptions?
I don't have special reason to distinguish these two.
Your suggestion look more reasonable for the exception naming.
Last, for `StateStoreMigratedException`, I would add that a user need to
> rediscover the store and cannot blindly retry as the store handle is
> invalid and a new store handle must be retrieved. That is a difference
> to `StreamThreadRebalancingException` that allows for "blind" retries
> that either resolve (if the store is still on the same instance after
> rebalancing finishes, or changes to `StateStoreMigratedException` if the
> store was migrated away during rebalancing).
>
Nice, it's great! Thank you.
The KIP already updated, please take a look. :)
On Wed, Oct 23, 2019 at 1:48 PM Matthias J. Sax <ma...@confluent.io>
wrote:
> Any update on this KIP?
>
> On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> > Sorry for the late reply. The 2.4 deadline kept us quite busy.
> >
> > About `StateStoreMigratedException`:
> >
> > Why is it only thrown if the state is REBALANCING? A store might be
> > migrated during a rebalance, and Kafka Streams might resume back to
> > RUNNING state and afterward somebody tries to use an old store handle.
> > Also, if state is REBALANCING, should we throw
> > `StreamThreadRebalancingException`? Hence, I think
> > `StateStoreMigratedException` does only make sense during `RUNNING`
> state.
> >
> >
> > Why do we need to distinguish between `KafkaStreamsNotRunningException`
> > and `StateStoreNotAvailableException`?
> >
> >
> > Last, why do we distinguish between `KafkaStreams` instance and
> > `StreamsThread`? To me, it seems we should always refer to the instance,
> > because that is the level of granularity in which we enable/disable IQ
> atm.
> >
> >
> > Last, for `StateStoreMigratedException`, I would add that a user need to
> > rediscover the store and cannot blindly retry as the store handle is
> > invalid and a new store handle must be retrieved. That is a difference
> > to `StreamThreadRebalancingException` that allows for "blind" retries
> > that either resolve (if the store is still on the same instance after
> > rebalancing finishes, or changes to `StateStoreMigratedException` if the
> > store was migrated away during rebalancing).
> >
> >
> >
> > -Matthias
> >
> > On 8/9/19 10:20 AM, Vito Jeng wrote:
> >> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9>
> >> <https://shorturl.at/CDNT9> seems incorrect.
> >>
> >> Please use the following instead: https://shorturl.at/bkKQU
> >>
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> >>
> >>> Thanks, Matthias!
> >>>
> >>>> About `StreamThreadNotStartedException`:
> >>>
> >>> Thank you for explanation. I agree with your opinion.
> >>> `CompositeReadOnlyXxxStore#get()` would never throw
> >>> `StreamThreadNotStartedException`.
> >>>
> >>> For the case that corresponding thread crashes after we handed out the
> >>> store handle. We may throw `KafkaStreamsNotRunningException` or
> >>> `StateStoreMigratedException`.
> >>> In `StreamThreadStateStoreProvider`, we would throw
> >>> `KafkaStreamsNotRunningException` when stream thread is not running(
> >>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
> >>> store is closed(https://shorturl.at/hrvAN). So I think we do not need
> to
> >>> add a new type for this case. Does that make sense?
> >>>
> >>>
> >>>> About `KafkaStreamsNotRunningException` vs
> >>> `StreamThreadNotRunningException`:
> >>>
> >>> I understand your point. I rename `StreamThreadNotRunningException` to
> >>> `KafkaStreamsNotRunningException`.
> >>>
> >>>
> >>> About check unknown state store names:
> >>> Thank you for the hint. I add a new type `UnknownStateStoreException`
> for
> >>> this case.
> >>>
> >>>
> >>>> Also, we should still have fatal exception
> >>> `StateStoreNotAvailableException`? Not sure why you remove it?
> >>>
> >>> Thank you point this, already add it again.
> >>>
> >>> The KIP already updated, please take a look.
> >>>
> >>> ---
> >>> Vito
> >>>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Any update on this KIP?
On 10/7/19 3:35 PM, Matthias J. Sax wrote:
> Sorry for the late reply. The 2.4 deadline kept us quite busy.
>
> About `StateStoreMigratedException`:
>
> Why is it only thrown if the state is REBALANCING? A store might be
> migrated during a rebalance, and Kafka Streams might resume back to
> RUNNING state and afterward somebody tries to use an old store handle.
> Also, if state is REBALANCING, should we throw
> `StreamThreadRebalancingException`? Hence, I think
> `StateStoreMigratedException` does only make sense during `RUNNING` state.
>
>
> Why do we need to distinguish between `KafkaStreamsNotRunningException`
> and `StateStoreNotAvailableException`?
>
>
> Last, why do we distinguish between `KafkaStreams` instance and
> `StreamsThread`? To me, it seems we should always refer to the instance,
> because that is the level of granularity in which we enable/disable IQ atm.
>
>
> Last, for `StateStoreMigratedException`, I would add that a user need to
> rediscover the store and cannot blindly retry as the store handle is
> invalid and a new store handle must be retrieved. That is a difference
> to `StreamThreadRebalancingException` that allows for "blind" retries
> that either resolve (if the store is still on the same instance after
> rebalancing finishes, or changes to `StateStoreMigratedException` if the
> store was migrated away during rebalancing).
>
>
>
> -Matthias
>
> On 8/9/19 10:20 AM, Vito Jeng wrote:
>> My bad. The short link `https://shorturl.at/CDNT9`
>> <https://shorturl.at/CDNT9> seems incorrect.
>>
>> Please use the following instead: https://shorturl.at/bkKQU
>>
>>
>> ---
>> Vito
>>
>>
>> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>>
>>> Thanks, Matthias!
>>>
>>>> About `StreamThreadNotStartedException`:
>>>
>>> Thank you for explanation. I agree with your opinion.
>>> `CompositeReadOnlyXxxStore#get()` would never throw
>>> `StreamThreadNotStartedException`.
>>>
>>> For the case that corresponding thread crashes after we handed out the
>>> store handle. We may throw `KafkaStreamsNotRunningException` or
>>> `StateStoreMigratedException`.
>>> In `StreamThreadStateStoreProvider`, we would throw
>>> `KafkaStreamsNotRunningException` when stream thread is not running(
>>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
>>> store is closed(https://shorturl.at/hrvAN). So I think we do not need to
>>> add a new type for this case. Does that make sense?
>>>
>>>
>>>> About `KafkaStreamsNotRunningException` vs
>>> `StreamThreadNotRunningException`:
>>>
>>> I understand your point. I rename `StreamThreadNotRunningException` to
>>> `KafkaStreamsNotRunningException`.
>>>
>>>
>>> About check unknown state store names:
>>> Thank you for the hint. I add a new type `UnknownStateStoreException` for
>>> this case.
>>>
>>>
>>>> Also, we should still have fatal exception
>>> `StateStoreNotAvailableException`? Not sure why you remove it?
>>>
>>> Thank you point this, already add it again.
>>>
>>> The KIP already updated, please take a look.
>>>
>>> ---
>>> Vito
>>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Sorry for the late reply. The 2.4 deadline kept us quite busy.
About `StateStoreMigratedException`:
Why is it only thrown if the state is REBALANCING? A store might be
migrated during a rebalance, and Kafka Streams might resume back to
RUNNING state and afterward somebody tries to use an old store handle.
Also, if state is REBALANCING, should we throw
`StreamThreadRebalancingException`? Hence, I think
`StateStoreMigratedException` does only make sense during `RUNNING` state.
Why do we need to distinguish between `KafkaStreamsNotRunningException`
and `StateStoreNotAvailableException`?
Last, why do we distinguish between `KafkaStreams` instance and
`StreamsThread`? To me, it seems we should always refer to the instance,
because that is the level of granularity in which we enable/disable IQ atm.
Last, for `StateStoreMigratedException`, I would add that a user need to
rediscover the store and cannot blindly retry as the store handle is
invalid and a new store handle must be retrieved. That is a difference
to `StreamThreadRebalancingException` that allows for "blind" retries
that either resolve (if the store is still on the same instance after
rebalancing finishes, or changes to `StateStoreMigratedException` if the
store was migrated away during rebalancing).
-Matthias
On 8/9/19 10:20 AM, Vito Jeng wrote:
> My bad. The short link `https://shorturl.at/CDNT9`
> <https://shorturl.at/CDNT9> seems incorrect.
>
> Please use the following instead: https://shorturl.at/bkKQU
>
>
> ---
> Vito
>
>
> On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw> wrote:
>
>> Thanks, Matthias!
>>
>>> About `StreamThreadNotStartedException`:
>>
>> Thank you for explanation. I agree with your opinion.
>> `CompositeReadOnlyXxxStore#get()` would never throw
>> `StreamThreadNotStartedException`.
>>
>> For the case that corresponding thread crashes after we handed out the
>> store handle. We may throw `KafkaStreamsNotRunningException` or
>> `StateStoreMigratedException`.
>> In `StreamThreadStateStoreProvider`, we would throw
>> `KafkaStreamsNotRunningException` when stream thread is not running(
>> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
>> store is closed(https://shorturl.at/hrvAN). So I think we do not need to
>> add a new type for this case. Does that make sense?
>>
>>
>>> About `KafkaStreamsNotRunningException` vs
>> `StreamThreadNotRunningException`:
>>
>> I understand your point. I rename `StreamThreadNotRunningException` to
>> `KafkaStreamsNotRunningException`.
>>
>>
>> About check unknown state store names:
>> Thank you for the hint. I add a new type `UnknownStateStoreException` for
>> this case.
>>
>>
>>> Also, we should still have fatal exception
>> `StateStoreNotAvailableException`? Not sure why you remove it?
>>
>> Thank you point this, already add it again.
>>
>> The KIP already updated, please take a look.
>>
>> ---
>> Vito
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
My bad. The short link `https://shorturl.at/CDNT9`
<https://shorturl.at/CDNT9> seems incorrect.
Please use the following instead: https://shorturl.at/bkKQU
---
Vito
On Fri, Aug 9, 2019 at 10:53 AM Vito Jeng <vi...@is-land.com.tw> wrote:
> Thanks, Matthias!
>
> > About `StreamThreadNotStartedException`:
>
> Thank you for explanation. I agree with your opinion.
> `CompositeReadOnlyXxxStore#get()` would never throw
> `StreamThreadNotStartedException`.
>
> For the case that corresponding thread crashes after we handed out the
> store handle. We may throw `KafkaStreamsNotRunningException` or
> `StateStoreMigratedException`.
> In `StreamThreadStateStoreProvider`, we would throw
> `KafkaStreamsNotRunningException` when stream thread is not running(
> https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
> store is closed(https://shorturl.at/hrvAN). So I think we do not need to
> add a new type for this case. Does that make sense?
>
>
> > About `KafkaStreamsNotRunningException` vs
> `StreamThreadNotRunningException`:
>
> I understand your point. I rename `StreamThreadNotRunningException` to
> `KafkaStreamsNotRunningException`.
>
>
> About check unknown state store names:
> Thank you for the hint. I add a new type `UnknownStateStoreException` for
> this case.
>
>
> > Also, we should still have fatal exception
> `StateStoreNotAvailableException`? Not sure why you remove it?
>
> Thank you point this, already add it again.
>
> The KIP already updated, please take a look.
>
> ---
> Vito
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Thanks, Matthias!
> About `StreamThreadNotStartedException`:
Thank you for explanation. I agree with your opinion.
`CompositeReadOnlyXxxStore#get()` would never throw
`StreamThreadNotStartedException`.
For the case that corresponding thread crashes after we handed out the
store handle. We may throw `KafkaStreamsNotRunningException` or
`StateStoreMigratedException`.
In `StreamThreadStateStoreProvider`, we would throw
`KafkaStreamsNotRunningException` when stream thread is not running(
https://shorturl.at/CDNT9) or throw `StateStoreMigratedException` when
store is closed(https://shorturl.at/hrvAN). So I think we do not need to
add a new type for this case. Does that make sense?
> About `KafkaStreamsNotRunningException` vs
`StreamThreadNotRunningException`:
I understand your point. I rename `StreamThreadNotRunningException` to
`KafkaStreamsNotRunningException`.
About check unknown state store names:
Thank you for the hint. I add a new type `UnknownStateStoreException` for
this case.
> Also, we should still have fatal exception
`StateStoreNotAvailableException`? Not sure why you remove it?
Thank you point this, already add it again.
The KIP already updated, please take a look.
---
Vito
On Tue, Jul 30, 2019 at 6:04 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Any update on this KIP Vito?
>
>
> On 7/11/19 4:26 PM, Matthias J. Sax wrote:
> > Thanks Vito! I think the KIP shapes out nicely!
> >
> >
> > To answer the open question you raised (I also adjust my answers based
> > on the latest KIP update)
> >
> >
> >
> > About `StreamThreadNotStartedException`: I understand what you pointed
> > out. However, I think we can consider the following: If a thread is not
> > started yet, and `KafkaStreams#store()` throw this exception, we would
> > not return a `CompositeReadOnlyXxxStore` to the user. Hence, `get()`
> > cannot be called. And if we return `CompositeReadOnlyXxxStore` the
> > thread was started and `get()` would never hit the condition to throw
> > the exception? Or do I miss something (this part of the logic is a
> > little tricky...)
> >
> > However, thinking about it, what could happen IMHO is, that the
> > corresponding thread crashes after we handed out the store handle. For
> > this case, it would make sense to throw an exception from `get()` but it
> > would be a different one IMHO. Maybe we need a new type
> > (`StreamThreadDeadException` or similar?) or we should reuse
> > `StoreMigratedException` because if a thread dies we would migrate the
> > store to another thread. (The tricky part might be, to detect this
> > condition correctly -- not 100% sure atm how we could do this.)
> >
> > What do you think about this?
> >
> >
> >
> > About `KafkaStreamsNotRunningException` vs
> > `StreamThreadNotRunningException` -- I see your point. Atm, I think we
> > don't allow querying at all if KafkaStreams is not in state RUNNING
> > (correct me if I am wrong). Hence, if there is an instance with 2
> > thread, and 1 thread is actually up and ready, but the other thread is
> > not, you cannot query anything. Only if both threads are in state
> > RUNNING we allow to query. It might be possible to change the code to
> > allow querying if a thread is ready independent from the other threads.
> > For this case, the name you suggest would make more sense. But I
> > _think_, that the current behavior is different and thus,
> > `KafkaStreamsNotRunningException` seems to reflect the current behavior
> > better? -- I also want to add that we are talking about a fatal
> > exception -- if a thread crashes, we would migrate the store to another
> > thread and it would not be fatal, but the store can be re-discovered.
> > Only if all thread would die, if would be fatal -- however, for this
> > case KafakStreams would transit to DEAD anyway.
> >
> >
> >
> >> When the user passes a store name to `KafkaStreams#store()`, does there
> >> have a way that distinguish the store name is "a wrong name" or
> "migrated"
> >> during `QueryableStoreProvider#getStore()`?
> >> From my current understanding, I cannot distinguish these two.
> >
> > This should be possible. In the private KafkaStreams constructor, we
> > have access to `InternalTopologyBuilder` that can give us all known
> > store names. Hence, we can get a set of all known store names, keep them
> > as a member variable and use in `KafkaStreams#store()` in an initial
> > check if the store name is valid or not.
> >
> >
> >
> >> Should we remove `StreamThreadNotRunningException` and throw
> >> `FatalStateStoreException` directly ?
> >
> > I would keep both, because `FatalStateStoreException` is not very
> > descriptive. Also, we should still have fatal exception
> > `StateStoreNotAvailableException`? Not sure why you remove it?
> >
> >
> >
> > Glad you found a way to avoid
> > `QueryableStoreType#setStreams(KafkaStreams streams)`.
> >
> >
> >
> > -Matthias
> >
> >
> > On 7/5/19 8:03 AM, Vito Jeng wrote:
> >> Hi, Mattias,
> >>
> >> Just completed the modification of KIP, please take a look when you are
> >> available.
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Wed, Jul 3, 2019 at 9:07 PM Vito Jeng <vi...@is-land.com.tw> wrote:
> >>
> >>> Hi, Matthias,
> >>>
> >>> This is second part.
> >>>
> >>>> For the internal exceptions:
> >>>>
> >>>> `StateStoreClosedException` -- why can it be wrapped as
> >>>> `StreamThreadNotStartedException` ? It seems that the later would only
> >>>> be thrown by `KafkaStreams#store()` and thus would be throw directly.
> >>>
> >>> Both `StateStoreClosedException` and `EmptyStateStoreException` not
> can be
> >>> wrapped as `StreamThreadNotStartedException`.
> >>> This is a mistaken written in the previous KIP. Thank you point this.
> >>>
> >>>> A closed-exception should only happen after a store was successfully
> >>>> retrieved but cannot be queried any longer? Hence, converting/wrapping
> >>>> it into a `StateStoreMigratedException` make sense. I am also not
> sure,
> >>>> when a closed-exception would be wrapped by a
> >>>> `StateStoreNotAvailableException` (implying my understanding as
> describe
> >>>> above)?
> >>>>
> >>>> Same questions about `EmptyStateStoreException`.
> >>>>
> >>>> Thinking about both internal exceptions twice, I am wondering if it
> >>>> makes sense to have both internal exceptions at all? I have the
> >>>> impression that it make only sense to wrap them with a
> >>>> `StateStoreMigragedException`, but if they are wrapped into the same
> >>>> exception all the time, we can just remove both and throw
> >>>> `StateStoreMigratedException` directly?
> >>>
> >>> After deeper thinking, I think you are right. It seems we can throw
> >>> `StateStoreMigratedException` directly.
> >>> So that we can remove `StateStoreClosedException`,
> >>> `EmptyStateStoreException` and `StateStoreNotAvailableException`.
> >>> Will update the KIP.
> >>>
> >>> BTW, if we remove above three exceptions, the
> >>> `StreamThreadNotRunningException` will be the only one sub class
> extends
> >>> from FatalStateStoreException.
> >>> Should we remove `StreamThreadNotRunningException` and throw
> >>> `FatalStateStoreException` directly ?
> >>>
> >>>> Last point: Why do we need to add?
> >>>> QueryableStoreType#setStreams(KafkaStreams streams);
> >>>> John asked this question already and you replied to it. But I am not
> >>>> sure what your answer means. Can you explain it in more detail?
> >>>
> >>> The main purpose is to pass the KafkaStreams reference into
> >>> CompositeReadOnlyKeyValueStore / CompositeReadOnlySessionStore/
> >>> CompositeReadOnlyWindowStore instance.
> >>> We need check KafkaStreams state to warp InvalidStateStoreException in
> to
> >>> other exception(e.g., StateStoreMigratedException) when the user
> accesses
> >>> these read-only stores.
> >>>
> >>> The original thought is to add `setStreams` method in to
> >>> QueryableStoreType. But now I think I find a better way during recent
> days.
> >>> This way does not need to change any public interface. So we can skip
> this
> >>> question. :)
> >>>
> >>>
> >>> I will update the KIP based on our discussion.
> >>> Thank you for help to finish the KIP!
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Thu, Jun 6, 2019 at 8:23 AM Matthias J. Sax <ma...@confluent.io>
> >>> wrote:
> >>>
> >>>> Hi Vito,
> >>>>
> >>>> sorry for dropping this discussion on the floor a while back. I was
> just
> >>>> re-reading the KIP and discussion thread, and I think it is shaping
> out
> >>>> nicely!
> >>>>
> >>>> I like the overall hierarchy of the exception classes.
> >>>>
> >>>> Some things are still not 100% clear:
> >>>>
> >>>>
> >>>> You listed all methods that may throw an `InvalidStateStoreException`
> >>>> atm. For the new exceptions, can any exception be thrown by any
> method?
> >>>> It might help to understand this relationship better.
> >>>>
> >>>> For example, StreamThreadNotStartedException, seems to only make sense
> >>>> for `KafkaStreams#store()`?
> >>>>
> >>>>
> >>>> For `StreamThreadNotRunningException` should we rename it to
> >>>> `KafkaStreamsNotRunningException` ?
> >>>>
> >>>>
> >>>> The description of `StreamThreadNotRunningException` and
> >>>> `StateStoreNotAvailableException` seems to be the same? From my
> >>>> understandng, the description makes sense for
> >>>> `StreamThreadNotRunningException` -- for
> >>>> `StateStoreNotAvailableException` I was expecting/inferring from the
> >>>> name, that it would be thrown if no such store exists in the topology
> at
> >>>> all (ie, user passed in a invalid/wrong store name). For this case,
> this
> >>>> exception should be thrown only from `KafkaStreams#store()` ?
> >>>>
> >>>>
> >>>> For the internal exceptions:
> >>>>
> >>>> `StateStoreClosedException` -- why can it be wrapped as
> >>>> `StreamThreadNotStartedException` ? It seems that the later would only
> >>>> be thrown by `KafkaStreams#store()` and thus would be throw directly.
> A
> >>>> closed-exception should only happen after a store was successfully
> >>>> retrieved but cannot be queried any longer? Hence, converting/wrapping
> >>>> it into a `StateStoreMigratedException` make sense. I am also not
> sure,
> >>>> when a closed-exception would be wrapped by a
> >>>> `StateStoreNotAvailableException` (implying my understanding as
> describe
> >>>> above)?
> >>>>
> >>>> Same questions about `EmptyStateStoreException`.
> >>>>
> >>>> Thinking about both internal exceptions twice, I am wondering if it
> >>>> makes sense to have both internal exceptions at all? I have the
> >>>> impression that it make only sense to wrap them with a
> >>>> `StateStoreMigragedException`, but if they are wrapped into the same
> >>>> exception all the time, we can just remove both and throw
> >>>> `StateStoreMigratedException` directly?
> >>>>
> >>>>
> >>>> Last point: Why do we need to add?
> >>>>
> >>>>> QueryableStoreType#setStreams(KafkaStreams streams);
> >>>>
> >>>> John asked this question already and you replied to it. But I am not
> >>>> sure what your answer means. Can you explain it in more detail?
> >>>>
> >>>>
> >>>>
> >>>> Thanks for your patience on this KIP!
> >>>>
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 11/11/18 4:55 AM, Vito Jeng wrote:
> >>>>> Hi, Matthias,
> >>>>>
> >>>>> KIP already updated.
> >>>>>
> >>>>>> - StateStoreClosedException:
> >>>>>> will be wrapped to StateStoreMigratedException or
> >>>>> StateStoreNotAvailableException later.
> >>>>>> Can you clarify the cases (ie, when will it be wrapped with the one
> or
> >>>>> the other)?
> >>>>>
> >>>>> For example, in the
> implementation(CompositeReadOnlyKeyValueStore#get),
> >>>> we
> >>>>> get all stores first, and then call ReadOnlyKeyValueStore#get to get
> >>>> value
> >>>>> in every store iteration.
> >>>>>
> >>>>> When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
> >>>> will
> >>>>> be thrown if the state store is not open.
> >>>>> We need catch StateStoreClosedException and wrap it in different
> >>>> exception
> >>>>> type:
> >>>>> * If the stream's state is CREATED, we wrap
> StateStoreClosedException
> >>>>> with StreamThreadNotStartedException. User can retry until to
> RUNNING.
> >>>>> * If the stream's state is RUNNING / REBALANCING, the state store
> >>>> should
> >>>>> be migrated, we wrap StateStoreClosedException with
> >>>>> StateStoreMigratedException. User can rediscover the state store.
> >>>>> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR,
> the
> >>>>> stream thread is not available, we wrap StateStoreClosedException
> with
> >>>>> StateStoreNotAvailableException. User cannot retry when this
> exception
> >>>> is
> >>>>> thrown.
> >>>>>
> >>>>>
> >>>>>> - StateStoreIsEmptyException:
> >>>>>> I don't understand the semantic of this exception. Maybe it's a
> naming
> >>>>> issue?
> >>>>>
> >>>>> I think yes. :)
> >>>>> Does `EmptyStateStoreException` is better ? (already updated in the
> KIP)
> >>>>>
> >>>>>
> >>>>>> - StateStoreIsEmptyException:
> >>>>>> will be wrapped to StateStoreMigratedException or
> >>>>> StateStoreNotAvailableException later.
> >>>>>> Also, can you clarify the cases (ie, when will it be wrapped with
> the
> >>>> one
> >>>>> or the other)?
> >>>>>
> >>>>> For example, in the implementation
> >>>> (CompositeReadOnlyKeyValueStore#get), we
> >>>>> call StateStoreProvider#stores (WrappingStoreProvider#stores) to get
> all
> >>>>> stores. EmptyStateStoreException will be thrown when cannot find any
> >>>> store
> >>>>> and then we need catch it and wrap it in different exception type:
> >>>>> * If the stream's state is CREATED, we wrap
> EmptyStateStoreException
> >>>> with
> >>>>> StreamThreadNotStartedException. User can retry until to RUNNING.
> >>>>> * If the stream's state is RUNNING / REBALANCING, the state store
> >>>> should
> >>>>> be migrated, we wrap EmptyStateStoreException with
> >>>>> StateStoreMigratedException. User can rediscover the state store.
> >>>>> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR,
> the
> >>>>> stream thread is not available, we wrap EmptyStateStoreException with
> >>>>> StateStoreNotAvailableException. User cannot retry when this
> exception
> >>>> is
> >>>>> thrown.
> >>>>>
> >>>>> I hope the above reply can clarify.
> >>>>>
> >>>>> The last one that was not replied was:
> >>>>>
> >>>>>> I am also wondering, if we should introduce a fatal exception
> >>>>>> `UnkownStateStoreException` to tell users that they passed in an
> >>>> unknown
> >>>>>> store name?
> >>>>>
> >>>>> Until now, unknown state store is not thinking about in the KIP.
> >>>>> I believe it would be very useful for users.
> >>>>>
> >>>>> Looking at the related code(WrappingStoreProvider#stores),
> >>>>> I found that I can't distinguish between the state store was migrated
> >>>> or an
> >>>>> unknown state store.
> >>>>>
> >>>>> Any thoughts?
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw>
> wrote:
> >>>>>
> >>>>>> Hi, Matthias,
> >>>>>>
> >>>>>> Sorry for the late reply.
> >>>>>>
> >>>>>>> I am wondering what the semantic impact/change is, if we introduce
> >>>>>>> `RetryableStateStoreException` and `FatalStateStoreException` that
> >>>> both
> >>>>>>> inherit from it. While I like the introduction of both from a high
> >>>> level
> >>>>>>> point of view, I just want to make sure it's semantically sound and
> >>>>>>> backward compatible. Atm, I think it's fine, but I want to point it
> >>>> out
> >>>>>>> such that everybody can think about this, too, so we can verify
> that
> >>>>>>> it's a natural evolving API change.
> >>>>>>
> >>>>>> Thank you for pointing this out. This's really important for public
> >>>> API.
> >>>>>>
> >>>>>> Just when I was replying to you, I found that KIP needs some modify.
> >>>>>> I will fix it ASAP, and then let's continue the discussion.
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <
> matthias@confluent.io>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hey Vito,
> >>>>>>>
> >>>>>>> I saw that you updated your PR, but did not reply to my last
> comments.
> >>>>>>> Any thoughts?
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
> >>>>>>>> Glad to have you back Vito :)
> >>>>>>>>
> >>>>>>>> Some follow up thoughts:
> >>>>>>>>
> >>>>>>>> - the current `InvalidStateStoreException` is documents as being
> >>>>>>>> sometimes retry-able. From the JavaDocs:
> >>>>>>>>
> >>>>>>>>> These exceptions may be transient [...] Hence, it is valid to
> >>>> backoff
> >>>>>>> and retry when handling this exception.
> >>>>>>>>
> >>>>>>>> I am wondering what the semantic impact/change is, if we introduce
> >>>>>>>> `RetryableStateStoreException` and `FatalStateStoreException` that
> >>>> both
> >>>>>>>> inherit from it. While I like the introduction of both from a high
> >>>> level
> >>>>>>>> point of view, I just want to make sure it's semantically sound
> and
> >>>>>>>> backward compatible. Atm, I think it's fine, but I want to point
> it
> >>>> out
> >>>>>>>> such that everybody can think about this, too, so we can verify
> that
> >>>>>>>> it's a natural evolving API change.
> >>>>>>>>
> >>>>>>>> - StateStoreClosedException:
> >>>>>>>>
> >>>>>>>>> will be wrapped to StateStoreMigratedException or
> >>>>>>> StateStoreNotAvailableException later.
> >>>>>>>>
> >>>>>>>> Can you clarify the cases (ie, when will it be wrapped with the
> one
> >>>> or
> >>>>>>>> the other)?
> >>>>>>>>
> >>>>>>>> - StateStoreIsEmptyException:
> >>>>>>>>
> >>>>>>>> I don't understand the semantic of this exception. Maybe it's a
> >>>> naming
> >>>>>>>> issue?
> >>>>>>>>
> >>>>>>>>> will be wrapped to StateStoreMigratedException or
> >>>>>>> StateStoreNotAvailableException later.
> >>>>>>>>
> >>>>>>>> Also, can you clarify the cases (ie, when will it be wrapped with
> the
> >>>>>>>> one or the other)?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I am also wondering, if we should introduce a fatal exception
> >>>>>>>> `UnkownStateStoreException` to tell users that they passed in an
> >>>> unknown
> >>>>>>>> store name?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 10/17/18 8:14 PM, vito jeng wrote:
> >>>>>>>>> Just open a PR for further discussion:
> >>>>>>>>> https://github.com/apache/kafka/pull/5814
> >>>>>>>>>
> >>>>>>>>> Any suggestion is welcome.
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>> Vito
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi John,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for reviewing the KIP.
> >>>>>>>>>>
> >>>>>>>>>>> I didn't follow the addition of a new method to the
> >>>>>>> QueryableStoreType
> >>>>>>>>>>> interface. Can you elaborate why this is necessary to support
> the
> >>>> new
> >>>>>>>>>>> exception types?
> >>>>>>>>>>
> >>>>>>>>>> To support the new exception types, I would check stream state
> in
> >>>> the
> >>>>>>>>>> following classes:
> >>>>>>>>>> - CompositeReadOnlyKeyValueStore class
> >>>>>>>>>> - CompositeReadOnlySessionStore class
> >>>>>>>>>> - CompositeReadOnlyWindowStore class
> >>>>>>>>>> - DelegatingPeekingKeyValueIterator class
> >>>>>>>>>>
> >>>>>>>>>> It is also necessary to keep backward compatibility. So I plan
> >>>> passing
> >>>>>>>>>> stream
> >>>>>>>>>> instance to QueryableStoreType instance during
> KafkaStreams#store()
> >>>>>>>>>> invoked.
> >>>>>>>>>> It looks a most simple way, I think.
> >>>>>>>>>>
> >>>>>>>>>> It is why I add a new method to the QueryableStoreType
> interface. I
> >>>>>>> can
> >>>>>>>>>> understand
> >>>>>>>>>> that we should try to avoid adding the public api method.
> However,
> >>>> at
> >>>>>>> the
> >>>>>>>>>> moment
> >>>>>>>>>> I have no better ideas.
> >>>>>>>>>>
> >>>>>>>>>> Any thoughts?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Also, looking over your KIP again, it seems valuable to
> introduce
> >>>>>>>>>>> "retriable store exception" and "fatal store exception" marker
> >>>>>>> interfaces
> >>>>>>>>>>> that the various exceptions can mix in. It would be nice from a
> >>>>>>> usability
> >>>>>>>>>>> perspective to be able to just log and retry on any "retriable"
> >>>>>>> exception
> >>>>>>>>>>> and log and shutdown on any fatal exception.
> >>>>>>>>>>
> >>>>>>>>>> I agree that this is valuable to the user.
> >>>>>>>>>> I'll update the KIP.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Thanks
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> Vito
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm glad to hear you're well again!
> >>>>>>>>>>>
> >>>>>>>>>>> I didn't follow the addition of a new method to the
> >>>>>>> QueryableStoreType
> >>>>>>>>>>> interface. Can you elaborate why this is necessary to support
> the
> >>>> new
> >>>>>>>>>>> exception types?
> >>>>>>>>>>>
> >>>>>>>>>>> Also, looking over your KIP again, it seems valuable to
> introduce
> >>>>>>>>>>> "retriable store exception" and "fatal store exception" marker
> >>>>>>> interfaces
> >>>>>>>>>>> that the various exceptions can mix in. It would be nice from a
> >>>>>>> usability
> >>>>>>>>>>> perspective to be able to just log and retry on any "retriable"
> >>>>>>> exception
> >>>>>>>>>>> and log and shutdown on any fatal exception.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> -John
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <
> wangguoz@gmail.com
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the explanation, that makes sense.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
> >>>>>>> matthias@confluent.io
> >>>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> The scenario I had I mind was, that KS is started in one
> thread
> >>>>>>> while
> >>>>>>>>>>> a
> >>>>>>>>>>>>> second thread has a reference to the object to issue queries.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If a query is issue before the "main thread" started KS, and
> the
> >>>>>>>>>>> "query
> >>>>>>>>>>>>> thread" knows that it will eventually get started, it can
> >>>> retry. On
> >>>>>>>>>>> the
> >>>>>>>>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
> >>>>>>>>>>> impossible
> >>>>>>>>>>>>> to issue any query against it now or in the future and thus
> the
> >>>>>>> error
> >>>>>>>>>>> is
> >>>>>>>>>>>>> not retryable.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
> >>>>>>>>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
> >>>>>>>>>>>>>> StreamThreadNotRunning, because I think users' handling
> logic
> >>>> for
> >>>>>>>>>>> the
> >>>>>>>>>>>>> third
> >>>>>>>>>>>>>> case would be likely the same as the second. Do you have
> some
> >>>>>>>>>>> scenarios
> >>>>>>>>>>>>>> where users may want to handle them differently?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
> >>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sorry to hear! Get well soon!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel
> free
> >>>> to
> >>>>>>>>>>> pick
> >>>>>>>>>>>> it
> >>>>>>>>>>>>>>> up again when you find time.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> >>>>>>> error?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
> >>>>>>> retryable
> >>>>>>>>>>>>> error.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state
> is
> >>>> not
> >>>>>>>>>>>>>>> RUNNING. The
> >>>>>>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I see. If this is the intention, than I would suggest to
> have
> >>>> two
> >>>>>>>>>>> (or
> >>>>>>>>>>>>>>> maybe three) different exceptions:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - StreamThreadRebalancingException (retryable)
> >>>>>>>>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in
> state
> >>>>>>>>>>>>>>> PENDING_SHUTDOWN or DEAD
> >>>>>>>>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The last one is tricky and could also be merged into one of
> >>>> the
> >>>>>>>>>>> first
> >>>>>>>>>>>>>>> two, depending if you want to argue that it's retryable or
> >>>> not.
> >>>>>>>>>>> (Just
> >>>>>>>>>>>>>>> food for though -- not sure what others think.)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you for your assistance.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Unfortunately, there is no further progress.
> >>>>>>>>>>>>>>>> About seven weeks ago, I was injured in sports. I had a
> >>>> broken
> >>>>>>>>>>> wrist
> >>>>>>>>>>>> on
> >>>>>>>>>>>>>>>> my left wrist.
> >>>>>>>>>>>>>>>> Many jobs are affected, including this KIP and
> >>>> implementation.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I just re-read it, and have a couple of follow up
> comments.
> >>>> Why
> >>>>>>>>>>> do
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also,
> do we
> >>>>>>>>>>> really
> >>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
> >>>>>>> instead
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> wrapping it later?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I think you may be right. As I say in the previous:
> >>>>>>>>>>>>>>>> "The original idea is that we can distinguish different
> state
> >>>>>>>>>>> store
> >>>>>>>>>>>>>>>> exception for different handling. But to be honest, I am
> not
> >>>>>>> quite
> >>>>>>>>>>>> sure
> >>>>>>>>>>>>>>>> this is necessary. Maybe have some change during
> >>>>>>> implementation."
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> During the implementation, I also feel we maybe not need
> >>>> wrapper
> >>>>>>>>>>> it.
> >>>>>>>>>>>>>>>> We can just throw the correctly directly.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> >>>> error?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
> >>>> retryable
> >>>>>>>>>>>> error.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
> >>>> not
> >>>>>>>>>>>>> RUNNING.
> >>>>>>>>>>>>>>> The
> >>>>>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
> >>>>>>> semantics
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>> unclear to me atm.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When the state is RUNNING, is
> `StateStoreClosedException` a
> >>>>>>>>>>>> retryable
> >>>>>>>>>>>>>>>> error?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> These two comments will be answered in another mail.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
> >>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I just re-read it, and have a couple of follow up
> comments.
> >>>> Why
> >>>>>>>>>>> do
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also,
> do we
> >>>>>>>>>>> really
> >>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
> >>>>>>> instead
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> wrapping it later?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
> >>>>>>> semantics
> >>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> unclear to me atm.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> >>>> error?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> When the state is RUNNING, is
> `StateStoreClosedException` a
> >>>>>>>>>>>> retryable
> >>>>>>>>>>>>>>>>> error?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method
> #fetch(K
> >>>>>>> key,
> >>>>>>>>>>>> long
> >>>>>>>>>>>>>>>>> time); that should be added
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Overall I like the KIP but some details are still
> unclear.
> >>>>>>> Maybe
> >>>>>>>>>>> it
> >>>>>>>>>>>>>>>>> might help if you open an PR in parallel?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>> Hi, Guozhang,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi, Bill,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'll try to make some update to make the KIP better.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
> >>>>>>> bbejeck@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> At this point, the only thing I would change is
> possibly
> >>>>>>>>>>> removing
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> listing of all methods called by the user and the
> listing
> >>>> of
> >>>>>>>>>>> all
> >>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>> types and focus on what states result in which
> exceptions
> >>>>>>>>>>> thrown
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>> Bill
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
> >>>>>>>>>>>> wangguoz@gmail.com>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me.
> >>>> I'm +1
> >>>>>>>>>>> on
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> About the base class InvalidStateStoreException
> itself,
> >>>> I'd
> >>>>>>>>>>>>> actually
> >>>>>>>>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as
> >>>> part
> >>>>>>> of
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> public
> >>>>>>>>>>>>>>>>>>>> API, for people who do not want to handle these cases
> >>>>>>>>>>> differently
> >>>>>>>>>>>>> (if
> >>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all
> >>>> three
> >>>>>>>>>>>>>>> exceptions
> >>>>>>>>>>>>>>>>>>>> one-by-one).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
> >>>>>>>>>>> john@confluent.io
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I think it's much nicer to give callers different
> >>>>>>> exceptions
> >>>>>>>>>>> to
> >>>>>>>>>>>>> tell
> >>>>>>>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>>>>>> whether the state store got migrated, whether it's
> still
> >>>>>>>>>>>>>>> initializing,
> >>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> whether there's some unrecoverable error.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
> >>>>>>>>>>>>> non-user-facing
> >>>>>>>>>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>>>>>> such as what exceptions we will throw internally. The
> >>>> KIP
> >>>>>>> is
> >>>>>>>>>>>>>>> primarily
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> discuss public interface changes.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> You might consider simply removing all the internal
> >>>> details
> >>>>>>>>>>> from
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>>>>>>> which will have the dual advantage that it makes the
> KIP
> >>>>>>>>>>> smaller
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in
> the
> >>>>>>>>>>> internal
> >>>>>>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>>>>>> when it comes to implementation.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I like your decision to have your refined exceptions
> >>>> extend
> >>>>>>>>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
> >>>>>>> compatibility.
> >>>>>>>>>>>> Since
> >>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> want
> >>>>>>>>>>>>>>>>>>>>> to encourage callers to catch the more specific
> >>>> exceptions,
> >>>>>>>>>>> and
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
> >>>>>>>>>>> anymore,
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
> >>>>>>>>>>>>>>>>>>> InvalidStateStoreException.
> >>>>>>>>>>>>>>>>>>>>> This will gently encourage callers to migrate to the
> new
> >>>>>>>>>>>> exception
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>> open
> >>>>>>>>>>>>>>>>>>>>> the possibility of removing
> InvalidStateStoreException
> >>>>>>>>>>> entirely
> >>>>>>>>>>>>> in a
> >>>>>>>>>>>>>>>>>>>> future
> >>>>>>>>>>>>>>>>>>>>> release.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions
> >>>> explicit?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback!
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the
> KIP or
> >>>>>>> not.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Got it!
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
> >>>> used?
> >>>>>>> Do
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
> >>>> all
> >>>>>>>>>>>> anymore,
> >>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
> >>>>>>> understanding).
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is
> >>>> not
> >>>>>>>>>>> thrown
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>>> anymore,
> >>>>>>>>>>>>>>>>>>>>>>> but only new sub-classes.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as
> category
> >>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> For example,
> `StreamThreadStateStoreProvider#stores()`
> >>>>>>> will
> >>>>>>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal
> exception).
> >>>>>>>>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
> >>>>>>>>>>>>>>>>>>>>>>> `StateStoreRetryableException` or
> >>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>> during
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
> >>>>>>> exceptions".
> >>>>>>>>>>>> It's
> >>>>>>>>>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is
> to
> >>>>>>>>>>>>> distinguish
> >>>>>>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
> >>>>>>>>>>>>>>>>>>>>>>> The original idea is that we can distinguish
> different
> >>>>>>>>>>> state
> >>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>>>>> exception for
> >>>>>>>>>>>>>>>>>>>>>>> different handling.
> >>>>>>>>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is
> >>>> necessary.
> >>>>>>>>>>> Maybe
> >>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>>>>>> some change during implementation.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Does it make sense?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for the update Vito!
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the
> KIP or
> >>>>>>> not.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
> >>>> used?
> >>>>>>> Do
> >>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
> >>>> all
> >>>>>>>>>>>> anymore,
> >>>>>>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
> >>>>>>> understanding).
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as
> category
> >>>>>>>>>>> exception
> >>>>>>>>>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
> >>>>>>> exceptions".
> >>>>>>>>>>>> It's
> >>>>>>>>>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review.
> >>>>>>>>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the
> long
> >>>>>>>>>>> pause...
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of
> the
> >>>>>>> code.
> >>>>>>>>>>>> It's
> >>>>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are
> >>>> affected.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a
> KIP.)
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
> >>>>>>>>>>> `Changes
> >>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>> call
> >>>>>>>>>>>>>>>>>>>>>>>> trace`.
> >>>>>>>>>>>>>>>>>>>>>>>>> Do you think it should be removed?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your
> KIP,
> >>>> the
> >>>>>>>>>>> main
> >>>>>>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
> >>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
> >>>>>>> things.
> >>>>>>>>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions
> >>>> extend
> >>>>>>>>>>> from
> >>>>>>>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Some questions:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
> >>>>>>>>>>> `InvalidStateStoreException`
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>>>>> this purpose?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Does this question miss some word?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> - What the superclass of
> >>>>>>>>>>> `StateStoreStreamThreadNotRunni
> >>>>>>>>>>>>>>>>>>>>> ngException`
> >>>>>>>>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
> >>>>>>> exception
> >>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried
> to
> >>>>>>> modify
> >>>>>>>>>>>> some
> >>>>>>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>> The modification is now complete. Please look
> again.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J.
> Sax <
> >>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the
> >>>> state
> >>>>>>>>>>> both
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams
> >>>> thread
> >>>>>>>>>>> and
> >>>>>>>>>>>>> state
> >>>>>>>>>>>>>>>>>>>>> store,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J.
> >>>> Sax <
> >>>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and
> thus, is
> >>>>>>> not
> >>>>>>>>>>>>> hosted
> >>>>>>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance
> is in
> >>>>>>>>>>> state
> >>>>>>>>>>>>>>>>>>>> rebalance
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover
> >>>> the
> >>>>>>>>>>> store.
> >>>>>>>>>>>>> For
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> second
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
> >>>>>>> finished.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
> >>>>>>>>>>> PENDING_SHUTDOWN, or
> >>>>>>>>>>>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they
> cannot
> >>>>>>>>>>>> rediscover
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means
> >>>> the
> >>>>>>>>>>> user
> >>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>>> plain
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or
> >>>> PENDING_SHUTDOWN
> >>>>>>> or
> >>>>>>>>>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance,
> the
> >>>>>>> user
> >>>>>>>>>>>> needs
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> rediscover
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias
> J.
> >>>> Sax
> >>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The
> original
> >>>>>>>>>>> idea of
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> Jira
> >>>>>>>>>>>>>>>>>>>>>>>> was,
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
> >>>>>>> "recovery"
> >>>>>>>>>>>>>>>>>>>> strategies
> >>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently
> >>>> recreated,
> >>>>>>> a
> >>>>>>>>>>>> user
> >>>>>>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> wait
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the
> other
> >>>>>>> hand,
> >>>>>>>>>>> if
> >>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> migrated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to
> >>>> rediscover
> >>>>>>>>>>> the
> >>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>>>>>>> instead
> >>>>>>>>>>>>>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section
> that
> >>>>>>>>>>> talks
> >>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>> ideas
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted
> to
> >>>>>>> figure
> >>>>>>>>>>>> out
> >>>>>>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify
> the
> >>>>>>>>>>> users, if
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>>>>>> wants
> >>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then
> >>>> probably
> >>>>>>> we
> >>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>> rename
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>>>>>>>>>>>>>>>>>>> StateStoreClosedException
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what
> are
> >>>> the
> >>>>>>>>>>>>> possible
> >>>>>>>>>>>>>>>>>>>>>> internal
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in
> the
> >>>>>>> call
> >>>>>>>>>>>>> trace,
> >>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what
> others,
> >>>>>>> and
> >>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>> ones
> >>>>>>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which
> ones
> >>>>>>>>>>> should
> >>>>>>>>>>>> not
> >>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>> wrapped
> >>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito
> jeng <
> >>>>>>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
> >>>>>>>>>>>>> confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>>>>>>>>>>>>>>>>>>> different+errors
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>> -- Guozhang
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> -- Guozhang
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Any update on this KIP Vito?
On 7/11/19 4:26 PM, Matthias J. Sax wrote:
> Thanks Vito! I think the KIP shapes out nicely!
>
>
> To answer the open question you raised (I also adjust my answers based
> on the latest KIP update)
>
>
>
> About `StreamThreadNotStartedException`: I understand what you pointed
> out. However, I think we can consider the following: If a thread is not
> started yet, and `KafkaStreams#store()` throw this exception, we would
> not return a `CompositeReadOnlyXxxStore` to the user. Hence, `get()`
> cannot be called. And if we return `CompositeReadOnlyXxxStore` the
> thread was started and `get()` would never hit the condition to throw
> the exception? Or do I miss something (this part of the logic is a
> little tricky...)
>
> However, thinking about it, what could happen IMHO is, that the
> corresponding thread crashes after we handed out the store handle. For
> this case, it would make sense to throw an exception from `get()` but it
> would be a different one IMHO. Maybe we need a new type
> (`StreamThreadDeadException` or similar?) or we should reuse
> `StoreMigratedException` because if a thread dies we would migrate the
> store to another thread. (The tricky part might be, to detect this
> condition correctly -- not 100% sure atm how we could do this.)
>
> What do you think about this?
>
>
>
> About `KafkaStreamsNotRunningException` vs
> `StreamThreadNotRunningException` -- I see your point. Atm, I think we
> don't allow querying at all if KafkaStreams is not in state RUNNING
> (correct me if I am wrong). Hence, if there is an instance with 2
> thread, and 1 thread is actually up and ready, but the other thread is
> not, you cannot query anything. Only if both threads are in state
> RUNNING we allow to query. It might be possible to change the code to
> allow querying if a thread is ready independent from the other threads.
> For this case, the name you suggest would make more sense. But I
> _think_, that the current behavior is different and thus,
> `KafkaStreamsNotRunningException` seems to reflect the current behavior
> better? -- I also want to add that we are talking about a fatal
> exception -- if a thread crashes, we would migrate the store to another
> thread and it would not be fatal, but the store can be re-discovered.
> Only if all thread would die, if would be fatal -- however, for this
> case KafakStreams would transit to DEAD anyway.
>
>
>
>> When the user passes a store name to `KafkaStreams#store()`, does there
>> have a way that distinguish the store name is "a wrong name" or "migrated"
>> during `QueryableStoreProvider#getStore()`?
>> From my current understanding, I cannot distinguish these two.
>
> This should be possible. In the private KafkaStreams constructor, we
> have access to `InternalTopologyBuilder` that can give us all known
> store names. Hence, we can get a set of all known store names, keep them
> as a member variable and use in `KafkaStreams#store()` in an initial
> check if the store name is valid or not.
>
>
>
>> Should we remove `StreamThreadNotRunningException` and throw
>> `FatalStateStoreException` directly ?
>
> I would keep both, because `FatalStateStoreException` is not very
> descriptive. Also, we should still have fatal exception
> `StateStoreNotAvailableException`? Not sure why you remove it?
>
>
>
> Glad you found a way to avoid
> `QueryableStoreType#setStreams(KafkaStreams streams)`.
>
>
>
> -Matthias
>
>
> On 7/5/19 8:03 AM, Vito Jeng wrote:
>> Hi, Mattias,
>>
>> Just completed the modification of KIP, please take a look when you are
>> available.
>>
>> ---
>> Vito
>>
>>
>> On Wed, Jul 3, 2019 at 9:07 PM Vito Jeng <vi...@is-land.com.tw> wrote:
>>
>>> Hi, Matthias,
>>>
>>> This is second part.
>>>
>>>> For the internal exceptions:
>>>>
>>>> `StateStoreClosedException` -- why can it be wrapped as
>>>> `StreamThreadNotStartedException` ? It seems that the later would only
>>>> be thrown by `KafkaStreams#store()` and thus would be throw directly.
>>>
>>> Both `StateStoreClosedException` and `EmptyStateStoreException` not can be
>>> wrapped as `StreamThreadNotStartedException`.
>>> This is a mistaken written in the previous KIP. Thank you point this.
>>>
>>>> A closed-exception should only happen after a store was successfully
>>>> retrieved but cannot be queried any longer? Hence, converting/wrapping
>>>> it into a `StateStoreMigratedException` make sense. I am also not sure,
>>>> when a closed-exception would be wrapped by a
>>>> `StateStoreNotAvailableException` (implying my understanding as describe
>>>> above)?
>>>>
>>>> Same questions about `EmptyStateStoreException`.
>>>>
>>>> Thinking about both internal exceptions twice, I am wondering if it
>>>> makes sense to have both internal exceptions at all? I have the
>>>> impression that it make only sense to wrap them with a
>>>> `StateStoreMigragedException`, but if they are wrapped into the same
>>>> exception all the time, we can just remove both and throw
>>>> `StateStoreMigratedException` directly?
>>>
>>> After deeper thinking, I think you are right. It seems we can throw
>>> `StateStoreMigratedException` directly.
>>> So that we can remove `StateStoreClosedException`,
>>> `EmptyStateStoreException` and `StateStoreNotAvailableException`.
>>> Will update the KIP.
>>>
>>> BTW, if we remove above three exceptions, the
>>> `StreamThreadNotRunningException` will be the only one sub class extends
>>> from FatalStateStoreException.
>>> Should we remove `StreamThreadNotRunningException` and throw
>>> `FatalStateStoreException` directly ?
>>>
>>>> Last point: Why do we need to add?
>>>> QueryableStoreType#setStreams(KafkaStreams streams);
>>>> John asked this question already and you replied to it. But I am not
>>>> sure what your answer means. Can you explain it in more detail?
>>>
>>> The main purpose is to pass the KafkaStreams reference into
>>> CompositeReadOnlyKeyValueStore / CompositeReadOnlySessionStore/
>>> CompositeReadOnlyWindowStore instance.
>>> We need check KafkaStreams state to warp InvalidStateStoreException in to
>>> other exception(e.g., StateStoreMigratedException) when the user accesses
>>> these read-only stores.
>>>
>>> The original thought is to add `setStreams` method in to
>>> QueryableStoreType. But now I think I find a better way during recent days.
>>> This way does not need to change any public interface. So we can skip this
>>> question. :)
>>>
>>>
>>> I will update the KIP based on our discussion.
>>> Thank you for help to finish the KIP!
>>>
>>> ---
>>> Vito
>>>
>>>
>>> On Thu, Jun 6, 2019 at 8:23 AM Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Hi Vito,
>>>>
>>>> sorry for dropping this discussion on the floor a while back. I was just
>>>> re-reading the KIP and discussion thread, and I think it is shaping out
>>>> nicely!
>>>>
>>>> I like the overall hierarchy of the exception classes.
>>>>
>>>> Some things are still not 100% clear:
>>>>
>>>>
>>>> You listed all methods that may throw an `InvalidStateStoreException`
>>>> atm. For the new exceptions, can any exception be thrown by any method?
>>>> It might help to understand this relationship better.
>>>>
>>>> For example, StreamThreadNotStartedException, seems to only make sense
>>>> for `KafkaStreams#store()`?
>>>>
>>>>
>>>> For `StreamThreadNotRunningException` should we rename it to
>>>> `KafkaStreamsNotRunningException` ?
>>>>
>>>>
>>>> The description of `StreamThreadNotRunningException` and
>>>> `StateStoreNotAvailableException` seems to be the same? From my
>>>> understandng, the description makes sense for
>>>> `StreamThreadNotRunningException` -- for
>>>> `StateStoreNotAvailableException` I was expecting/inferring from the
>>>> name, that it would be thrown if no such store exists in the topology at
>>>> all (ie, user passed in a invalid/wrong store name). For this case, this
>>>> exception should be thrown only from `KafkaStreams#store()` ?
>>>>
>>>>
>>>> For the internal exceptions:
>>>>
>>>> `StateStoreClosedException` -- why can it be wrapped as
>>>> `StreamThreadNotStartedException` ? It seems that the later would only
>>>> be thrown by `KafkaStreams#store()` and thus would be throw directly. A
>>>> closed-exception should only happen after a store was successfully
>>>> retrieved but cannot be queried any longer? Hence, converting/wrapping
>>>> it into a `StateStoreMigratedException` make sense. I am also not sure,
>>>> when a closed-exception would be wrapped by a
>>>> `StateStoreNotAvailableException` (implying my understanding as describe
>>>> above)?
>>>>
>>>> Same questions about `EmptyStateStoreException`.
>>>>
>>>> Thinking about both internal exceptions twice, I am wondering if it
>>>> makes sense to have both internal exceptions at all? I have the
>>>> impression that it make only sense to wrap them with a
>>>> `StateStoreMigragedException`, but if they are wrapped into the same
>>>> exception all the time, we can just remove both and throw
>>>> `StateStoreMigratedException` directly?
>>>>
>>>>
>>>> Last point: Why do we need to add?
>>>>
>>>>> QueryableStoreType#setStreams(KafkaStreams streams);
>>>>
>>>> John asked this question already and you replied to it. But I am not
>>>> sure what your answer means. Can you explain it in more detail?
>>>>
>>>>
>>>>
>>>> Thanks for your patience on this KIP!
>>>>
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 11/11/18 4:55 AM, Vito Jeng wrote:
>>>>> Hi, Matthias,
>>>>>
>>>>> KIP already updated.
>>>>>
>>>>>> - StateStoreClosedException:
>>>>>> will be wrapped to StateStoreMigratedException or
>>>>> StateStoreNotAvailableException later.
>>>>>> Can you clarify the cases (ie, when will it be wrapped with the one or
>>>>> the other)?
>>>>>
>>>>> For example, in the implementation(CompositeReadOnlyKeyValueStore#get),
>>>> we
>>>>> get all stores first, and then call ReadOnlyKeyValueStore#get to get
>>>> value
>>>>> in every store iteration.
>>>>>
>>>>> When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
>>>> will
>>>>> be thrown if the state store is not open.
>>>>> We need catch StateStoreClosedException and wrap it in different
>>>> exception
>>>>> type:
>>>>> * If the stream's state is CREATED, we wrap StateStoreClosedException
>>>>> with StreamThreadNotStartedException. User can retry until to RUNNING.
>>>>> * If the stream's state is RUNNING / REBALANCING, the state store
>>>> should
>>>>> be migrated, we wrap StateStoreClosedException with
>>>>> StateStoreMigratedException. User can rediscover the state store.
>>>>> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
>>>>> stream thread is not available, we wrap StateStoreClosedException with
>>>>> StateStoreNotAvailableException. User cannot retry when this exception
>>>> is
>>>>> thrown.
>>>>>
>>>>>
>>>>>> - StateStoreIsEmptyException:
>>>>>> I don't understand the semantic of this exception. Maybe it's a naming
>>>>> issue?
>>>>>
>>>>> I think yes. :)
>>>>> Does `EmptyStateStoreException` is better ? (already updated in the KIP)
>>>>>
>>>>>
>>>>>> - StateStoreIsEmptyException:
>>>>>> will be wrapped to StateStoreMigratedException or
>>>>> StateStoreNotAvailableException later.
>>>>>> Also, can you clarify the cases (ie, when will it be wrapped with the
>>>> one
>>>>> or the other)?
>>>>>
>>>>> For example, in the implementation
>>>> (CompositeReadOnlyKeyValueStore#get), we
>>>>> call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
>>>>> stores. EmptyStateStoreException will be thrown when cannot find any
>>>> store
>>>>> and then we need catch it and wrap it in different exception type:
>>>>> * If the stream's state is CREATED, we wrap EmptyStateStoreException
>>>> with
>>>>> StreamThreadNotStartedException. User can retry until to RUNNING.
>>>>> * If the stream's state is RUNNING / REBALANCING, the state store
>>>> should
>>>>> be migrated, we wrap EmptyStateStoreException with
>>>>> StateStoreMigratedException. User can rediscover the state store.
>>>>> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
>>>>> stream thread is not available, we wrap EmptyStateStoreException with
>>>>> StateStoreNotAvailableException. User cannot retry when this exception
>>>> is
>>>>> thrown.
>>>>>
>>>>> I hope the above reply can clarify.
>>>>>
>>>>> The last one that was not replied was:
>>>>>
>>>>>> I am also wondering, if we should introduce a fatal exception
>>>>>> `UnkownStateStoreException` to tell users that they passed in an
>>>> unknown
>>>>>> store name?
>>>>>
>>>>> Until now, unknown state store is not thinking about in the KIP.
>>>>> I believe it would be very useful for users.
>>>>>
>>>>> Looking at the related code(WrappingStoreProvider#stores),
>>>>> I found that I can't distinguish between the state store was migrated
>>>> or an
>>>>> unknown state store.
>>>>>
>>>>> Any thoughts?
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw> wrote:
>>>>>
>>>>>> Hi, Matthias,
>>>>>>
>>>>>> Sorry for the late reply.
>>>>>>
>>>>>>> I am wondering what the semantic impact/change is, if we introduce
>>>>>>> `RetryableStateStoreException` and `FatalStateStoreException` that
>>>> both
>>>>>>> inherit from it. While I like the introduction of both from a high
>>>> level
>>>>>>> point of view, I just want to make sure it's semantically sound and
>>>>>>> backward compatible. Atm, I think it's fine, but I want to point it
>>>> out
>>>>>>> such that everybody can think about this, too, so we can verify that
>>>>>>> it's a natural evolving API change.
>>>>>>
>>>>>> Thank you for pointing this out. This's really important for public
>>>> API.
>>>>>>
>>>>>> Just when I was replying to you, I found that KIP needs some modify.
>>>>>> I will fix it ASAP, and then let's continue the discussion.
>>>>>>
>>>>>> ---
>>>>>> Vito
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey Vito,
>>>>>>>
>>>>>>> I saw that you updated your PR, but did not reply to my last comments.
>>>>>>> Any thoughts?
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
>>>>>>>> Glad to have you back Vito :)
>>>>>>>>
>>>>>>>> Some follow up thoughts:
>>>>>>>>
>>>>>>>> - the current `InvalidStateStoreException` is documents as being
>>>>>>>> sometimes retry-able. From the JavaDocs:
>>>>>>>>
>>>>>>>>> These exceptions may be transient [...] Hence, it is valid to
>>>> backoff
>>>>>>> and retry when handling this exception.
>>>>>>>>
>>>>>>>> I am wondering what the semantic impact/change is, if we introduce
>>>>>>>> `RetryableStateStoreException` and `FatalStateStoreException` that
>>>> both
>>>>>>>> inherit from it. While I like the introduction of both from a high
>>>> level
>>>>>>>> point of view, I just want to make sure it's semantically sound and
>>>>>>>> backward compatible. Atm, I think it's fine, but I want to point it
>>>> out
>>>>>>>> such that everybody can think about this, too, so we can verify that
>>>>>>>> it's a natural evolving API change.
>>>>>>>>
>>>>>>>> - StateStoreClosedException:
>>>>>>>>
>>>>>>>>> will be wrapped to StateStoreMigratedException or
>>>>>>> StateStoreNotAvailableException later.
>>>>>>>>
>>>>>>>> Can you clarify the cases (ie, when will it be wrapped with the one
>>>> or
>>>>>>>> the other)?
>>>>>>>>
>>>>>>>> - StateStoreIsEmptyException:
>>>>>>>>
>>>>>>>> I don't understand the semantic of this exception. Maybe it's a
>>>> naming
>>>>>>>> issue?
>>>>>>>>
>>>>>>>>> will be wrapped to StateStoreMigratedException or
>>>>>>> StateStoreNotAvailableException later.
>>>>>>>>
>>>>>>>> Also, can you clarify the cases (ie, when will it be wrapped with the
>>>>>>>> one or the other)?
>>>>>>>>
>>>>>>>>
>>>>>>>> I am also wondering, if we should introduce a fatal exception
>>>>>>>> `UnkownStateStoreException` to tell users that they passed in an
>>>> unknown
>>>>>>>> store name?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/17/18 8:14 PM, vito jeng wrote:
>>>>>>>>> Just open a PR for further discussion:
>>>>>>>>> https://github.com/apache/kafka/pull/5814
>>>>>>>>>
>>>>>>>>> Any suggestion is welcome.
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Vito
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi John,
>>>>>>>>>>
>>>>>>>>>> Thanks for reviewing the KIP.
>>>>>>>>>>
>>>>>>>>>>> I didn't follow the addition of a new method to the
>>>>>>> QueryableStoreType
>>>>>>>>>>> interface. Can you elaborate why this is necessary to support the
>>>> new
>>>>>>>>>>> exception types?
>>>>>>>>>>
>>>>>>>>>> To support the new exception types, I would check stream state in
>>>> the
>>>>>>>>>> following classes:
>>>>>>>>>> - CompositeReadOnlyKeyValueStore class
>>>>>>>>>> - CompositeReadOnlySessionStore class
>>>>>>>>>> - CompositeReadOnlyWindowStore class
>>>>>>>>>> - DelegatingPeekingKeyValueIterator class
>>>>>>>>>>
>>>>>>>>>> It is also necessary to keep backward compatibility. So I plan
>>>> passing
>>>>>>>>>> stream
>>>>>>>>>> instance to QueryableStoreType instance during KafkaStreams#store()
>>>>>>>>>> invoked.
>>>>>>>>>> It looks a most simple way, I think.
>>>>>>>>>>
>>>>>>>>>> It is why I add a new method to the QueryableStoreType interface. I
>>>>>>> can
>>>>>>>>>> understand
>>>>>>>>>> that we should try to avoid adding the public api method. However,
>>>> at
>>>>>>> the
>>>>>>>>>> moment
>>>>>>>>>> I have no better ideas.
>>>>>>>>>>
>>>>>>>>>> Any thoughts?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>>>>>>>>> "retriable store exception" and "fatal store exception" marker
>>>>>>> interfaces
>>>>>>>>>>> that the various exceptions can mix in. It would be nice from a
>>>>>>> usability
>>>>>>>>>>> perspective to be able to just log and retry on any "retriable"
>>>>>>> exception
>>>>>>>>>>> and log and shutdown on any fatal exception.
>>>>>>>>>>
>>>>>>>>>> I agree that this is valuable to the user.
>>>>>>>>>> I'll update the KIP.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>
>>>>>>>>>>> I'm glad to hear you're well again!
>>>>>>>>>>>
>>>>>>>>>>> I didn't follow the addition of a new method to the
>>>>>>> QueryableStoreType
>>>>>>>>>>> interface. Can you elaborate why this is necessary to support the
>>>> new
>>>>>>>>>>> exception types?
>>>>>>>>>>>
>>>>>>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>>>>>>>>> "retriable store exception" and "fatal store exception" marker
>>>>>>> interfaces
>>>>>>>>>>> that the various exceptions can mix in. It would be nice from a
>>>>>>> usability
>>>>>>>>>>> perspective to be able to just log and retry on any "retriable"
>>>>>>> exception
>>>>>>>>>>> and log and shutdown on any fatal exception.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> -John
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wangguoz@gmail.com
>>>>>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the explanation, that makes sense.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
>>>>>>> matthias@confluent.io
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The scenario I had I mind was, that KS is started in one thread
>>>>>>> while
>>>>>>>>>>> a
>>>>>>>>>>>>> second thread has a reference to the object to issue queries.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If a query is issue before the "main thread" started KS, and the
>>>>>>>>>>> "query
>>>>>>>>>>>>> thread" knows that it will eventually get started, it can
>>>> retry. On
>>>>>>>>>>> the
>>>>>>>>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>>>>>>>>>>> impossible
>>>>>>>>>>>>> to issue any query against it now or in the future and thus the
>>>>>>> error
>>>>>>>>>>> is
>>>>>>>>>>>>> not retryable.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
>>>>>>>>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
>>>>>>>>>>>>>> StreamThreadNotRunning, because I think users' handling logic
>>>> for
>>>>>>>>>>> the
>>>>>>>>>>>>> third
>>>>>>>>>>>>>> case would be likely the same as the second. Do you have some
>>>>>>>>>>> scenarios
>>>>>>>>>>>>>> where users may want to handle them differently?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry to hear! Get well soon!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free
>>>> to
>>>>>>>>>>> pick
>>>>>>>>>>>> it
>>>>>>>>>>>>>>> up again when you find time.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>>>>>>> error?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>>>>>>> retryable
>>>>>>>>>>>>> error.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
>>>> not
>>>>>>>>>>>>>>> RUNNING. The
>>>>>>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I see. If this is the intention, than I would suggest to have
>>>> two
>>>>>>>>>>> (or
>>>>>>>>>>>>>>> maybe three) different exceptions:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - StreamThreadRebalancingException (retryable)
>>>>>>>>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
>>>>>>>>>>>>>>> PENDING_SHUTDOWN or DEAD
>>>>>>>>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The last one is tricky and could also be merged into one of
>>>> the
>>>>>>>>>>> first
>>>>>>>>>>>>>>> two, depending if you want to argue that it's retryable or
>>>> not.
>>>>>>>>>>> (Just
>>>>>>>>>>>>>>> food for though -- not sure what others think.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you for your assistance.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Unfortunately, there is no further progress.
>>>>>>>>>>>>>>>> About seven weeks ago, I was injured in sports. I had a
>>>> broken
>>>>>>>>>>> wrist
>>>>>>>>>>>> on
>>>>>>>>>>>>>>>> my left wrist.
>>>>>>>>>>>>>>>> Many jobs are affected, including this KIP and
>>>> implementation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
>>>> Why
>>>>>>>>>>> do
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>>>>>>>>> really
>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>>>>>>> instead
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> wrapping it later?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think you may be right. As I say in the previous:
>>>>>>>>>>>>>>>> "The original idea is that we can distinguish different state
>>>>>>>>>>> store
>>>>>>>>>>>>>>>> exception for different handling. But to be honest, I am not
>>>>>>> quite
>>>>>>>>>>>> sure
>>>>>>>>>>>>>>>> this is necessary. Maybe have some change during
>>>>>>> implementation."
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> During the implementation, I also feel we maybe not need
>>>> wrapper
>>>>>>>>>>> it.
>>>>>>>>>>>>>>>> We can just throw the correctly directly.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>>>> error?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>>>> retryable
>>>>>>>>>>>> error.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
>>>> not
>>>>>>>>>>>>> RUNNING.
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>>>>>>> semantics
>>>>>>>>>>> is
>>>>>>>>>>>>>>>> unclear to me atm.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>>>>>>>>> retryable
>>>>>>>>>>>>>>>> error?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> These two comments will be answered in another mail.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
>>>> Why
>>>>>>>>>>> do
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>>>>>>>>> really
>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>>>>>>> instead
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>> wrapping it later?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>>>>>>> semantics
>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> unclear to me atm.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>>>> error?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>>>>>>>>> retryable
>>>>>>>>>>>>>>>>> error?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
>>>>>>> key,
>>>>>>>>>>>> long
>>>>>>>>>>>>>>>>> time); that should be added
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Overall I like the KIP but some details are still unclear.
>>>>>>> Maybe
>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> might help if you open an PR in parallel?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>> Hi, Guozhang,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi, Bill,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'll try to make some update to make the KIP better.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
>>>>>>> bbejeck@gmail.com
>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> At this point, the only thing I would change is possibly
>>>>>>>>>>> removing
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> listing of all methods called by the user and the listing
>>>> of
>>>>>>>>>>> all
>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>> types and focus on what states result in which exceptions
>>>>>>>>>>> thrown
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Bill
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>>>>>>>>>>>> wangguoz@gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me.
>>>> I'm +1
>>>>>>>>>>> on
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> About the base class InvalidStateStoreException itself,
>>>> I'd
>>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as
>>>> part
>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>>>> API, for people who do not want to handle these cases
>>>>>>>>>>> differently
>>>>>>>>>>>>> (if
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all
>>>> three
>>>>>>>>>>>>>>> exceptions
>>>>>>>>>>>>>>>>>>>> one-by-one).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>>>>>>>>>>> john@confluent.io
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think it's much nicer to give callers different
>>>>>>> exceptions
>>>>>>>>>>> to
>>>>>>>>>>>>> tell
>>>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>>>> whether the state store got migrated, whether it's still
>>>>>>>>>>>>>>> initializing,
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> whether there's some unrecoverable error.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
>>>>>>>>>>>>> non-user-facing
>>>>>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>>>>>> such as what exceptions we will throw internally. The
>>>> KIP
>>>>>>> is
>>>>>>>>>>>>>>> primarily
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> discuss public interface changes.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> You might consider simply removing all the internal
>>>> details
>>>>>>>>>>> from
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
>>>>>>>>>>> smaller
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>>>>>> when it comes to implementation.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I like your decision to have your refined exceptions
>>>> extend
>>>>>>>>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
>>>>>>> compatibility.
>>>>>>>>>>>> Since
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>>>>> to encourage callers to catch the more specific
>>>> exceptions,
>>>>>>>>>>> and
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
>>>>>>>>>>> anymore,
>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
>>>>>>>>>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
>>>>>>>>>>>> exception
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
>>>>>>>>>>> entirely
>>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>>>> future
>>>>>>>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions
>>>> explicit?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>>>>>>> not.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Got it!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
>>>> used?
>>>>>>> Do
>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
>>>> all
>>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>>>>>>> understanding).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is
>>>> not
>>>>>>>>>>> thrown
>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>>>>>>>>> but only new sub-classes.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
>>>>>>> will
>>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>>>>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
>>>>>>>>>>>>>>>>>>>>>>> `StateStoreRetryableException` or
>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>>>>>>> exceptions".
>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>>>>>>>>>>>>> distinguish
>>>>>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
>>>>>>>>>>>>>>>>>>>>>>> The original idea is that we can distinguish different
>>>>>>>>>>> state
>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>>> exception for
>>>>>>>>>>>>>>>>>>>>>>> different handling.
>>>>>>>>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is
>>>> necessary.
>>>>>>>>>>> Maybe
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>>> some change during implementation.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Does it make sense?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the update Vito!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>>>>>>> not.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
>>>> used?
>>>>>>> Do
>>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
>>>> all
>>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>>>>>>> understanding).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>>>>>>> exceptions".
>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review.
>>>>>>>>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>>>>>>>>>>> pause...
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
>>>>>>> code.
>>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are
>>>> affected.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
>>>>>>>>>>> `Changes
>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> call
>>>>>>>>>>>>>>>>>>>>>>>> trace`.
>>>>>>>>>>>>>>>>>>>>>>>>> Do you think it should be removed?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP,
>>>> the
>>>>>>>>>>> main
>>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
>>>>>>> things.
>>>>>>>>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions
>>>> extend
>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Some questions:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
>>>>>>>>>>> `InvalidStateStoreException`
>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>> this purpose?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Does this question miss some word?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - What the superclass of
>>>>>>>>>>> `StateStoreStreamThreadNotRunni
>>>>>>>>>>>>>>>>>>>>> ngException`
>>>>>>>>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
>>>>>>> exception
>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
>>>>>>> modify
>>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the
>>>> state
>>>>>>>>>>> both
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams
>>>> thread
>>>>>>>>>>> and
>>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>>>> store,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J.
>>>> Sax <
>>>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is
>>>>>>> not
>>>>>>>>>>>>> hosted
>>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover
>>>> the
>>>>>>>>>>> store.
>>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
>>>>>>> finished.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>>>>>>>>>>> PENDING_SHUTDOWN, or
>>>>>>>>>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>>>>>>>>>>>> rediscover
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means
>>>> the
>>>>>>>>>>> user
>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>>> plain
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or
>>>> PENDING_SHUTDOWN
>>>>>>> or
>>>>>>>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the
>>>>>>> user
>>>>>>>>>>>> needs
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> rediscover
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J.
>>>> Sax
>>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>>>>>>>>>>> idea of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> Jira
>>>>>>>>>>>>>>>>>>>>>>>> was,
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
>>>>>>> "recovery"
>>>>>>>>>>>>>>>>>>>> strategies
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently
>>>> recreated,
>>>>>>> a
>>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other
>>>>>>> hand,
>>>>>>>>>>> if
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to
>>>> rediscover
>>>>>>>>>>> the
>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>>>>>>>>>>> talks
>>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>> ideas
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
>>>>>>> figure
>>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>>>>>>>>>>> users, if
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then
>>>> probably
>>>>>>> we
>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>>>>>>>>>>>>>>>>>>> StateStoreClosedException
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are
>>>> the
>>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the
>>>>>>> call
>>>>>>>>>>>>> trace,
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others,
>>>>>>> and
>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>> ones
>>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>>>>>>>>>>> should
>>>>>>>>>>>> not
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>> wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>>>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>>>>>>>>>>>>> confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>>>>>>>>>>>>>>>>>>> different+errors
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks Vito! I think the KIP shapes out nicely!
To answer the open question you raised (I also adjust my answers based
on the latest KIP update)
About `StreamThreadNotStartedException`: I understand what you pointed
out. However, I think we can consider the following: If a thread is not
started yet, and `KafkaStreams#store()` throw this exception, we would
not return a `CompositeReadOnlyXxxStore` to the user. Hence, `get()`
cannot be called. And if we return `CompositeReadOnlyXxxStore` the
thread was started and `get()` would never hit the condition to throw
the exception? Or do I miss something (this part of the logic is a
little tricky...)
However, thinking about it, what could happen IMHO is, that the
corresponding thread crashes after we handed out the store handle. For
this case, it would make sense to throw an exception from `get()` but it
would be a different one IMHO. Maybe we need a new type
(`StreamThreadDeadException` or similar?) or we should reuse
`StoreMigratedException` because if a thread dies we would migrate the
store to another thread. (The tricky part might be, to detect this
condition correctly -- not 100% sure atm how we could do this.)
What do you think about this?
About `KafkaStreamsNotRunningException` vs
`StreamThreadNotRunningException` -- I see your point. Atm, I think we
don't allow querying at all if KafkaStreams is not in state RUNNING
(correct me if I am wrong). Hence, if there is an instance with 2
thread, and 1 thread is actually up and ready, but the other thread is
not, you cannot query anything. Only if both threads are in state
RUNNING we allow to query. It might be possible to change the code to
allow querying if a thread is ready independent from the other threads.
For this case, the name you suggest would make more sense. But I
_think_, that the current behavior is different and thus,
`KafkaStreamsNotRunningException` seems to reflect the current behavior
better? -- I also want to add that we are talking about a fatal
exception -- if a thread crashes, we would migrate the store to another
thread and it would not be fatal, but the store can be re-discovered.
Only if all thread would die, if would be fatal -- however, for this
case KafakStreams would transit to DEAD anyway.
> When the user passes a store name to `KafkaStreams#store()`, does there
> have a way that distinguish the store name is "a wrong name" or "migrated"
> during `QueryableStoreProvider#getStore()`?
> From my current understanding, I cannot distinguish these two.
This should be possible. In the private KafkaStreams constructor, we
have access to `InternalTopologyBuilder` that can give us all known
store names. Hence, we can get a set of all known store names, keep them
as a member variable and use in `KafkaStreams#store()` in an initial
check if the store name is valid or not.
> Should we remove `StreamThreadNotRunningException` and throw
> `FatalStateStoreException` directly ?
I would keep both, because `FatalStateStoreException` is not very
descriptive. Also, we should still have fatal exception
`StateStoreNotAvailableException`? Not sure why you remove it?
Glad you found a way to avoid
`QueryableStoreType#setStreams(KafkaStreams streams)`.
-Matthias
On 7/5/19 8:03 AM, Vito Jeng wrote:
> Hi, Mattias,
>
> Just completed the modification of KIP, please take a look when you are
> available.
>
> ---
> Vito
>
>
> On Wed, Jul 3, 2019 at 9:07 PM Vito Jeng <vi...@is-land.com.tw> wrote:
>
>> Hi, Matthias,
>>
>> This is second part.
>>
>>> For the internal exceptions:
>>>
>>> `StateStoreClosedException` -- why can it be wrapped as
>>> `StreamThreadNotStartedException` ? It seems that the later would only
>>> be thrown by `KafkaStreams#store()` and thus would be throw directly.
>>
>> Both `StateStoreClosedException` and `EmptyStateStoreException` not can be
>> wrapped as `StreamThreadNotStartedException`.
>> This is a mistaken written in the previous KIP. Thank you point this.
>>
>>> A closed-exception should only happen after a store was successfully
>>> retrieved but cannot be queried any longer? Hence, converting/wrapping
>>> it into a `StateStoreMigratedException` make sense. I am also not sure,
>>> when a closed-exception would be wrapped by a
>>> `StateStoreNotAvailableException` (implying my understanding as describe
>>> above)?
>>>
>>> Same questions about `EmptyStateStoreException`.
>>>
>>> Thinking about both internal exceptions twice, I am wondering if it
>>> makes sense to have both internal exceptions at all? I have the
>>> impression that it make only sense to wrap them with a
>>> `StateStoreMigragedException`, but if they are wrapped into the same
>>> exception all the time, we can just remove both and throw
>>> `StateStoreMigratedException` directly?
>>
>> After deeper thinking, I think you are right. It seems we can throw
>> `StateStoreMigratedException` directly.
>> So that we can remove `StateStoreClosedException`,
>> `EmptyStateStoreException` and `StateStoreNotAvailableException`.
>> Will update the KIP.
>>
>> BTW, if we remove above three exceptions, the
>> `StreamThreadNotRunningException` will be the only one sub class extends
>> from FatalStateStoreException.
>> Should we remove `StreamThreadNotRunningException` and throw
>> `FatalStateStoreException` directly ?
>>
>>> Last point: Why do we need to add?
>>> QueryableStoreType#setStreams(KafkaStreams streams);
>>> John asked this question already and you replied to it. But I am not
>>> sure what your answer means. Can you explain it in more detail?
>>
>> The main purpose is to pass the KafkaStreams reference into
>> CompositeReadOnlyKeyValueStore / CompositeReadOnlySessionStore/
>> CompositeReadOnlyWindowStore instance.
>> We need check KafkaStreams state to warp InvalidStateStoreException in to
>> other exception(e.g., StateStoreMigratedException) when the user accesses
>> these read-only stores.
>>
>> The original thought is to add `setStreams` method in to
>> QueryableStoreType. But now I think I find a better way during recent days.
>> This way does not need to change any public interface. So we can skip this
>> question. :)
>>
>>
>> I will update the KIP based on our discussion.
>> Thank you for help to finish the KIP!
>>
>> ---
>> Vito
>>
>>
>> On Thu, Jun 6, 2019 at 8:23 AM Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>
>>> Hi Vito,
>>>
>>> sorry for dropping this discussion on the floor a while back. I was just
>>> re-reading the KIP and discussion thread, and I think it is shaping out
>>> nicely!
>>>
>>> I like the overall hierarchy of the exception classes.
>>>
>>> Some things are still not 100% clear:
>>>
>>>
>>> You listed all methods that may throw an `InvalidStateStoreException`
>>> atm. For the new exceptions, can any exception be thrown by any method?
>>> It might help to understand this relationship better.
>>>
>>> For example, StreamThreadNotStartedException, seems to only make sense
>>> for `KafkaStreams#store()`?
>>>
>>>
>>> For `StreamThreadNotRunningException` should we rename it to
>>> `KafkaStreamsNotRunningException` ?
>>>
>>>
>>> The description of `StreamThreadNotRunningException` and
>>> `StateStoreNotAvailableException` seems to be the same? From my
>>> understandng, the description makes sense for
>>> `StreamThreadNotRunningException` -- for
>>> `StateStoreNotAvailableException` I was expecting/inferring from the
>>> name, that it would be thrown if no such store exists in the topology at
>>> all (ie, user passed in a invalid/wrong store name). For this case, this
>>> exception should be thrown only from `KafkaStreams#store()` ?
>>>
>>>
>>> For the internal exceptions:
>>>
>>> `StateStoreClosedException` -- why can it be wrapped as
>>> `StreamThreadNotStartedException` ? It seems that the later would only
>>> be thrown by `KafkaStreams#store()` and thus would be throw directly. A
>>> closed-exception should only happen after a store was successfully
>>> retrieved but cannot be queried any longer? Hence, converting/wrapping
>>> it into a `StateStoreMigratedException` make sense. I am also not sure,
>>> when a closed-exception would be wrapped by a
>>> `StateStoreNotAvailableException` (implying my understanding as describe
>>> above)?
>>>
>>> Same questions about `EmptyStateStoreException`.
>>>
>>> Thinking about both internal exceptions twice, I am wondering if it
>>> makes sense to have both internal exceptions at all? I have the
>>> impression that it make only sense to wrap them with a
>>> `StateStoreMigragedException`, but if they are wrapped into the same
>>> exception all the time, we can just remove both and throw
>>> `StateStoreMigratedException` directly?
>>>
>>>
>>> Last point: Why do we need to add?
>>>
>>>> QueryableStoreType#setStreams(KafkaStreams streams);
>>>
>>> John asked this question already and you replied to it. But I am not
>>> sure what your answer means. Can you explain it in more detail?
>>>
>>>
>>>
>>> Thanks for your patience on this KIP!
>>>
>>>
>>>
>>> -Matthias
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 11/11/18 4:55 AM, Vito Jeng wrote:
>>>> Hi, Matthias,
>>>>
>>>> KIP already updated.
>>>>
>>>>> - StateStoreClosedException:
>>>>> will be wrapped to StateStoreMigratedException or
>>>> StateStoreNotAvailableException later.
>>>>> Can you clarify the cases (ie, when will it be wrapped with the one or
>>>> the other)?
>>>>
>>>> For example, in the implementation(CompositeReadOnlyKeyValueStore#get),
>>> we
>>>> get all stores first, and then call ReadOnlyKeyValueStore#get to get
>>> value
>>>> in every store iteration.
>>>>
>>>> When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
>>> will
>>>> be thrown if the state store is not open.
>>>> We need catch StateStoreClosedException and wrap it in different
>>> exception
>>>> type:
>>>> * If the stream's state is CREATED, we wrap StateStoreClosedException
>>>> with StreamThreadNotStartedException. User can retry until to RUNNING.
>>>> * If the stream's state is RUNNING / REBALANCING, the state store
>>> should
>>>> be migrated, we wrap StateStoreClosedException with
>>>> StateStoreMigratedException. User can rediscover the state store.
>>>> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
>>>> stream thread is not available, we wrap StateStoreClosedException with
>>>> StateStoreNotAvailableException. User cannot retry when this exception
>>> is
>>>> thrown.
>>>>
>>>>
>>>>> - StateStoreIsEmptyException:
>>>>> I don't understand the semantic of this exception. Maybe it's a naming
>>>> issue?
>>>>
>>>> I think yes. :)
>>>> Does `EmptyStateStoreException` is better ? (already updated in the KIP)
>>>>
>>>>
>>>>> - StateStoreIsEmptyException:
>>>>> will be wrapped to StateStoreMigratedException or
>>>> StateStoreNotAvailableException later.
>>>>> Also, can you clarify the cases (ie, when will it be wrapped with the
>>> one
>>>> or the other)?
>>>>
>>>> For example, in the implementation
>>> (CompositeReadOnlyKeyValueStore#get), we
>>>> call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
>>>> stores. EmptyStateStoreException will be thrown when cannot find any
>>> store
>>>> and then we need catch it and wrap it in different exception type:
>>>> * If the stream's state is CREATED, we wrap EmptyStateStoreException
>>> with
>>>> StreamThreadNotStartedException. User can retry until to RUNNING.
>>>> * If the stream's state is RUNNING / REBALANCING, the state store
>>> should
>>>> be migrated, we wrap EmptyStateStoreException with
>>>> StateStoreMigratedException. User can rediscover the state store.
>>>> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
>>>> stream thread is not available, we wrap EmptyStateStoreException with
>>>> StateStoreNotAvailableException. User cannot retry when this exception
>>> is
>>>> thrown.
>>>>
>>>> I hope the above reply can clarify.
>>>>
>>>> The last one that was not replied was:
>>>>
>>>>> I am also wondering, if we should introduce a fatal exception
>>>>> `UnkownStateStoreException` to tell users that they passed in an
>>> unknown
>>>>> store name?
>>>>
>>>> Until now, unknown state store is not thinking about in the KIP.
>>>> I believe it would be very useful for users.
>>>>
>>>> Looking at the related code(WrappingStoreProvider#stores),
>>>> I found that I can't distinguish between the state store was migrated
>>> or an
>>>> unknown state store.
>>>>
>>>> Any thoughts?
>>>>
>>>> ---
>>>> Vito
>>>>
>>>>
>>>>
>>>> On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw> wrote:
>>>>
>>>>> Hi, Matthias,
>>>>>
>>>>> Sorry for the late reply.
>>>>>
>>>>>> I am wondering what the semantic impact/change is, if we introduce
>>>>>> `RetryableStateStoreException` and `FatalStateStoreException` that
>>> both
>>>>>> inherit from it. While I like the introduction of both from a high
>>> level
>>>>>> point of view, I just want to make sure it's semantically sound and
>>>>>> backward compatible. Atm, I think it's fine, but I want to point it
>>> out
>>>>>> such that everybody can think about this, too, so we can verify that
>>>>>> it's a natural evolving API change.
>>>>>
>>>>> Thank you for pointing this out. This's really important for public
>>> API.
>>>>>
>>>>> Just when I was replying to you, I found that KIP needs some modify.
>>>>> I will fix it ASAP, and then let's continue the discussion.
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>>
>>>>> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
>>>>> wrote:
>>>>>
>>>>>> Hey Vito,
>>>>>>
>>>>>> I saw that you updated your PR, but did not reply to my last comments.
>>>>>> Any thoughts?
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
>>>>>>> Glad to have you back Vito :)
>>>>>>>
>>>>>>> Some follow up thoughts:
>>>>>>>
>>>>>>> - the current `InvalidStateStoreException` is documents as being
>>>>>>> sometimes retry-able. From the JavaDocs:
>>>>>>>
>>>>>>>> These exceptions may be transient [...] Hence, it is valid to
>>> backoff
>>>>>> and retry when handling this exception.
>>>>>>>
>>>>>>> I am wondering what the semantic impact/change is, if we introduce
>>>>>>> `RetryableStateStoreException` and `FatalStateStoreException` that
>>> both
>>>>>>> inherit from it. While I like the introduction of both from a high
>>> level
>>>>>>> point of view, I just want to make sure it's semantically sound and
>>>>>>> backward compatible. Atm, I think it's fine, but I want to point it
>>> out
>>>>>>> such that everybody can think about this, too, so we can verify that
>>>>>>> it's a natural evolving API change.
>>>>>>>
>>>>>>> - StateStoreClosedException:
>>>>>>>
>>>>>>>> will be wrapped to StateStoreMigratedException or
>>>>>> StateStoreNotAvailableException later.
>>>>>>>
>>>>>>> Can you clarify the cases (ie, when will it be wrapped with the one
>>> or
>>>>>>> the other)?
>>>>>>>
>>>>>>> - StateStoreIsEmptyException:
>>>>>>>
>>>>>>> I don't understand the semantic of this exception. Maybe it's a
>>> naming
>>>>>>> issue?
>>>>>>>
>>>>>>>> will be wrapped to StateStoreMigratedException or
>>>>>> StateStoreNotAvailableException later.
>>>>>>>
>>>>>>> Also, can you clarify the cases (ie, when will it be wrapped with the
>>>>>>> one or the other)?
>>>>>>>
>>>>>>>
>>>>>>> I am also wondering, if we should introduce a fatal exception
>>>>>>> `UnkownStateStoreException` to tell users that they passed in an
>>> unknown
>>>>>>> store name?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/17/18 8:14 PM, vito jeng wrote:
>>>>>>>> Just open a PR for further discussion:
>>>>>>>> https://github.com/apache/kafka/pull/5814
>>>>>>>>
>>>>>>>> Any suggestion is welcome.
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi John,
>>>>>>>>>
>>>>>>>>> Thanks for reviewing the KIP.
>>>>>>>>>
>>>>>>>>>> I didn't follow the addition of a new method to the
>>>>>> QueryableStoreType
>>>>>>>>>> interface. Can you elaborate why this is necessary to support the
>>> new
>>>>>>>>>> exception types?
>>>>>>>>>
>>>>>>>>> To support the new exception types, I would check stream state in
>>> the
>>>>>>>>> following classes:
>>>>>>>>> - CompositeReadOnlyKeyValueStore class
>>>>>>>>> - CompositeReadOnlySessionStore class
>>>>>>>>> - CompositeReadOnlyWindowStore class
>>>>>>>>> - DelegatingPeekingKeyValueIterator class
>>>>>>>>>
>>>>>>>>> It is also necessary to keep backward compatibility. So I plan
>>> passing
>>>>>>>>> stream
>>>>>>>>> instance to QueryableStoreType instance during KafkaStreams#store()
>>>>>>>>> invoked.
>>>>>>>>> It looks a most simple way, I think.
>>>>>>>>>
>>>>>>>>> It is why I add a new method to the QueryableStoreType interface. I
>>>>>> can
>>>>>>>>> understand
>>>>>>>>> that we should try to avoid adding the public api method. However,
>>> at
>>>>>> the
>>>>>>>>> moment
>>>>>>>>> I have no better ideas.
>>>>>>>>>
>>>>>>>>> Any thoughts?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>>>>>>>> "retriable store exception" and "fatal store exception" marker
>>>>>> interfaces
>>>>>>>>>> that the various exceptions can mix in. It would be nice from a
>>>>>> usability
>>>>>>>>>> perspective to be able to just log and retry on any "retriable"
>>>>>> exception
>>>>>>>>>> and log and shutdown on any fatal exception.
>>>>>>>>>
>>>>>>>>> I agree that this is valuable to the user.
>>>>>>>>> I'll update the KIP.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Vito
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Vito,
>>>>>>>>>>
>>>>>>>>>> I'm glad to hear you're well again!
>>>>>>>>>>
>>>>>>>>>> I didn't follow the addition of a new method to the
>>>>>> QueryableStoreType
>>>>>>>>>> interface. Can you elaborate why this is necessary to support the
>>> new
>>>>>>>>>> exception types?
>>>>>>>>>>
>>>>>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>>>>>>>> "retriable store exception" and "fatal store exception" marker
>>>>>> interfaces
>>>>>>>>>> that the various exceptions can mix in. It would be nice from a
>>>>>> usability
>>>>>>>>>> perspective to be able to just log and retry on any "retriable"
>>>>>> exception
>>>>>>>>>> and log and shutdown on any fatal exception.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> -John
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wangguoz@gmail.com
>>>>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the explanation, that makes sense.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Guozhang
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
>>>>>> matthias@confluent.io
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The scenario I had I mind was, that KS is started in one thread
>>>>>> while
>>>>>>>>>> a
>>>>>>>>>>>> second thread has a reference to the object to issue queries.
>>>>>>>>>>>>
>>>>>>>>>>>> If a query is issue before the "main thread" started KS, and the
>>>>>>>>>> "query
>>>>>>>>>>>> thread" knows that it will eventually get started, it can
>>> retry. On
>>>>>>>>>> the
>>>>>>>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>>>>>>>>>> impossible
>>>>>>>>>>>> to issue any query against it now or in the future and thus the
>>>>>> error
>>>>>>>>>> is
>>>>>>>>>>>> not retryable.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>
>>>>>>>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
>>>>>>>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
>>>>>>>>>>>>> StreamThreadNotRunning, because I think users' handling logic
>>> for
>>>>>>>>>> the
>>>>>>>>>>>> third
>>>>>>>>>>>>> case would be likely the same as the second. Do you have some
>>>>>>>>>> scenarios
>>>>>>>>>>>>> where users may want to handle them differently?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry to hear! Get well soon!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free
>>> to
>>>>>>>>>> pick
>>>>>>>>>>> it
>>>>>>>>>>>>>> up again when you find time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>>>>>> error?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>>>>>> retryable
>>>>>>>>>>>> error.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
>>> not
>>>>>>>>>>>>>> RUNNING. The
>>>>>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I see. If this is the intention, than I would suggest to have
>>> two
>>>>>>>>>> (or
>>>>>>>>>>>>>> maybe three) different exceptions:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - StreamThreadRebalancingException (retryable)
>>>>>>>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
>>>>>>>>>>>>>> PENDING_SHUTDOWN or DEAD
>>>>>>>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The last one is tricky and could also be merged into one of
>>> the
>>>>>>>>>> first
>>>>>>>>>>>>>> two, depending if you want to argue that it's retryable or
>>> not.
>>>>>>>>>> (Just
>>>>>>>>>>>>>> food for though -- not sure what others think.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you for your assistance.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Unfortunately, there is no further progress.
>>>>>>>>>>>>>>> About seven weeks ago, I was injured in sports. I had a
>>> broken
>>>>>>>>>> wrist
>>>>>>>>>>> on
>>>>>>>>>>>>>>> my left wrist.
>>>>>>>>>>>>>>> Many jobs are affected, including this KIP and
>>> implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
>>> Why
>>>>>>>>>> do
>>>>>>>>>>> we
>>>>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>>>>>>>> really
>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>>>>>> instead
>>>>>>>>>> of
>>>>>>>>>>>>>>>> wrapping it later?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think you may be right. As I say in the previous:
>>>>>>>>>>>>>>> "The original idea is that we can distinguish different state
>>>>>>>>>> store
>>>>>>>>>>>>>>> exception for different handling. But to be honest, I am not
>>>>>> quite
>>>>>>>>>>> sure
>>>>>>>>>>>>>>> this is necessary. Maybe have some change during
>>>>>> implementation."
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> During the implementation, I also feel we maybe not need
>>> wrapper
>>>>>>>>>> it.
>>>>>>>>>>>>>>> We can just throw the correctly directly.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>>> error?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>>> retryable
>>>>>>>>>>> error.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
>>> not
>>>>>>>>>>>> RUNNING.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>>>>>> semantics
>>>>>>>>>> is
>>>>>>>>>>>>>>> unclear to me atm.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>>>>>>>> retryable
>>>>>>>>>>>>>>> error?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> These two comments will be answered in another mail.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
>>> Why
>>>>>>>>>> do
>>>>>>>>>>> we
>>>>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>>>>>>>> really
>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>>>>>> instead
>>>>>>>>>> of
>>>>>>>>>>>>>>>> wrapping it later?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>>>>>> semantics
>>>>>>>>>> is
>>>>>>>>>>>>>>>> unclear to me atm.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>>> error?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>>>>>>>> retryable
>>>>>>>>>>>>>>>> error?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
>>>>>> key,
>>>>>>>>>>> long
>>>>>>>>>>>>>>>> time); that should be added
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Overall I like the KIP but some details are still unclear.
>>>>>> Maybe
>>>>>>>>>> it
>>>>>>>>>>>>>>>> might help if you open an PR in parallel?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>> Hi, Guozhang,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi, Bill,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'll try to make some update to make the KIP better.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
>>>>>> bbejeck@gmail.com
>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At this point, the only thing I would change is possibly
>>>>>>>>>> removing
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> listing of all methods called by the user and the listing
>>> of
>>>>>>>>>> all
>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>> types and focus on what states result in which exceptions
>>>>>>>>>> thrown
>>>>>>>>>>> to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Bill
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>>>>>>>>>>> wangguoz@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me.
>>> I'm +1
>>>>>>>>>> on
>>>>>>>>>>> the
>>>>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> About the base class InvalidStateStoreException itself,
>>> I'd
>>>>>>>>>>>> actually
>>>>>>>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as
>>> part
>>>>>> of
>>>>>>>>>> the
>>>>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>>>>> API, for people who do not want to handle these cases
>>>>>>>>>> differently
>>>>>>>>>>>> (if
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all
>>> three
>>>>>>>>>>>>>> exceptions
>>>>>>>>>>>>>>>>>>> one-by-one).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>>>>>>>>>> john@confluent.io
>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think it's much nicer to give callers different
>>>>>> exceptions
>>>>>>>>>> to
>>>>>>>>>>>> tell
>>>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>>>> whether the state store got migrated, whether it's still
>>>>>>>>>>>>>> initializing,
>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> whether there's some unrecoverable error.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
>>>>>>>>>>>> non-user-facing
>>>>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>>>>> such as what exceptions we will throw internally. The
>>> KIP
>>>>>> is
>>>>>>>>>>>>>> primarily
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> discuss public interface changes.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> You might consider simply removing all the internal
>>> details
>>>>>>>>>> from
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
>>>>>>>>>> smaller
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
>>>>>>>>>> internal
>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>>>>> when it comes to implementation.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I like your decision to have your refined exceptions
>>> extend
>>>>>>>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
>>>>>> compatibility.
>>>>>>>>>>> Since
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>>>>> to encourage callers to catch the more specific
>>> exceptions,
>>>>>>>>>> and
>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
>>>>>>>>>> anymore,
>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
>>>>>>>>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
>>>>>>>>>>> exception
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
>>>>>>>>>> entirely
>>>>>>>>>>>> in a
>>>>>>>>>>>>>>>>>>> future
>>>>>>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions
>>> explicit?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>>>>>> not.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Got it!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
>>> used?
>>>>>> Do
>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
>>> all
>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>>>>>> understanding).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is
>>> not
>>>>>>>>>> thrown
>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>>>>>>>> but only new sub-classes.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
>>>>>> will
>>>>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>>>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
>>>>>>>>>>>>>>>>>>>>>> `StateStoreRetryableException` or
>>>>>> `StateStoreFailException`
>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>>>>>> exceptions".
>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>>>>>>>>>>>> distinguish
>>>>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
>>>>>>>>>>>>>>>>>>>>>> The original idea is that we can distinguish different
>>>>>>>>>> state
>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>>> exception for
>>>>>>>>>>>>>>>>>>>>>> different handling.
>>>>>>>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is
>>> necessary.
>>>>>>>>>> Maybe
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>>>>> some change during implementation.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Does it make sense?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thanks for the update Vito!
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>>>>>> not.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
>>> used?
>>>>>> Do
>>>>>>>>>>> you
>>>>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
>>> all
>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>>>>>> understanding).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>>>>>>>> exception
>>>>>>>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>>>>>> exceptions".
>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the review.
>>>>>>>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>>>>>>>>>> pause...
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
>>>>>> code.
>>>>>>>>>>> It's
>>>>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are
>>> affected.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
>>>>>>>>>> `Changes
>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> call
>>>>>>>>>>>>>>>>>>>>>>> trace`.
>>>>>>>>>>>>>>>>>>>>>>>> Do you think it should be removed?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP,
>>> the
>>>>>>>>>> main
>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
>>>>>> things.
>>>>>>>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions
>>> extend
>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Some questions:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
>>>>>>>>>> `InvalidStateStoreException`
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> this purpose?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Does this question miss some word?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - What the superclass of
>>>>>>>>>> `StateStoreStreamThreadNotRunni
>>>>>>>>>>>>>>>>>>>> ngException`
>>>>>>>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
>>>>>> exception
>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
>>>>>> modify
>>>>>>>>>>> some
>>>>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
>>>>>>>>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the
>>> state
>>>>>>>>>> both
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams
>>> thread
>>>>>>>>>> and
>>>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>>> store,
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J.
>>> Sax <
>>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is
>>>>>> not
>>>>>>>>>>>> hosted
>>>>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>>>>>>>>>> state
>>>>>>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover
>>> the
>>>>>>>>>> store.
>>>>>>>>>>>> For
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
>>>>>> finished.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>>>>>>>>>> PENDING_SHUTDOWN, or
>>>>>>>>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>>>>>>>>>>> rediscover
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means
>>> the
>>>>>>>>>> user
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>>> plain
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or
>>> PENDING_SHUTDOWN
>>>>>> or
>>>>>>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the
>>>>>> user
>>>>>>>>>>> needs
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> rediscover
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J.
>>> Sax
>>>>>> <
>>>>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>>>>>>>>>> idea of
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> Jira
>>>>>>>>>>>>>>>>>>>>>>> was,
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
>>>>>> "recovery"
>>>>>>>>>>>>>>>>>>> strategies
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently
>>> recreated,
>>>>>> a
>>>>>>>>>>> user
>>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other
>>>>>> hand,
>>>>>>>>>> if
>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to
>>> rediscover
>>>>>>>>>> the
>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>>>>>>>>>> talks
>>>>>>>>>>>> about
>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>> ideas
>>>>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
>>>>>> figure
>>>>>>>>>>> out
>>>>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>>>>>>>>>> users, if
>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then
>>> probably
>>>>>> we
>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>>>>>>>>>>>>>>>>>> StateStoreClosedException
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are
>>> the
>>>>>>>>>>>> possible
>>>>>>>>>>>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the
>>>>>> call
>>>>>>>>>>>> trace,
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others,
>>>>>> and
>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>> ones
>>>>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>>>>>>>>>> should
>>>>>>>>>>> not
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> wrapped
>>>>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>>>>>>>>>>>> confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>>>>>>>>>>>>>>>>>> different+errors
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, Mattias,
Sorry for late reply.
Nice to hear from you, thank you for re-reading the discussion thread and
help to finish the KIP.
I spend some time rereading our past discussions and updating the KIP.
Hope the following reply can have more clarification.
> For example, StreamThreadNotStartedException, seems to only make sense
for `KafkaStreams#store()`?
StreamThreadNotStartedException not only thrown from `KafkaStreams#store()`
but also thrown from all public methods in the `ReadOnlyKeyValueStore` and
`ReadOnlySessionStore` and `ReadOnlyWindowStore` (class
CompositeReadOnlyKeyValueStore, CompositeReadOnlySessionStore,
CompositeReadOnlyWindowStore).
For example, in the `CompositeReadOnlyKeyValueStore#get(key)`, the first
step is calling `StateStoreProvider#stores()` to get all matched stores.
This may cause `StreamThreadNotStartedException`. Basically, all public
methods in the `ReadOnlyKeyValueStore` and `ReadOnlySessionStore` and
`ReadOnlyWindowStore` need to call `StateStoreProvider#stores()`. This
could be considered the same situation as `KafkaStreams#store()` calling
`QueryableStoreProvider#getStore()`
Does this make sense?
> For `StreamThreadNotRunningException` should we rename it to
`KafkaStreamsNotRunningException` ?
Hmmm...I am not sure. `KafkaStreamsNotRunningException` is a good name and
I like it. KafkaStreams running implies that all stream threads are
running.
But `StreamThreadNotRunningException` could be exactly to tell the user
what happened, IMHO.
I personally pick `StreamThreadNotRunningException`. But I am fine to pick
`KafkaStreamsNotRunningException` if you think this name is better. :)
> The description of `StreamThreadNotRunningException` and
`StateStoreNotAvailableException` seems to be the same? From my
understandng, the description makes sense for
`StreamThreadNotRunningException` -- for
`StateStoreNotAvailableException` I was expecting/inferring from the
name, that it would be thrown if no such store exists in the topology at
all (ie, user passed in a invalid/wrong store name). For this case, this
exception should be thrown only from `KafkaStreams#store()` ?
Just like `StreamThreadNotStartedException`(my replied above), from my
current understanding,
the `StreamThreadNotRunningException` and `StateStoreNotAvailableException`
not only thrown from `KafkaStreams#store()` but also thrown from all public
methods in the `ReadOnlyKeyValueStore` and `ReadOnlySessionStore` and
`ReadOnlyWindowStore.
for `StateStoreNotAvailableException` -- It would be thrown if no such
store exists in the topology at all(EmptyStateStoreException) and would be
thrown when the store already closed(StateStoreClosedException). for
`StreamThreadNotRunningException` - It would be thrown if the stream thread
is not running.
IMHO, it would be better that distinguish the two different exception so
that user can handle by different way.
What do you think?
During I wrote this letter, I have a question and would like to ask you for
clarification:
When the user passes a store name to `KafkaStreams#store()`, does there
have a way that distinguish the store name is "a wrong name" or "migrated"
during `QueryableStoreProvider#getStore()`? (
https://github.com/apache/kafka/blob/11641c7f53b2ee156490b2aadf79552d266ec956/streams/src/main/java/org/apache/kafka/streams/state/internals/QueryableStoreProvider.java#L61-L63
) From my current understanding, I cannot distinguish these two.
I will reply with the rest of the questions in another letter.
Thank you again.
---
Vito
On Thu, Jun 6, 2019 at 8:23 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Hi Vito,
>
> sorry for dropping this discussion on the floor a while back. I was just
> re-reading the KIP and discussion thread, and I think it is shaping out
> nicely!
>
> I like the overall hierarchy of the exception classes.
>
> Some things are still not 100% clear:
>
>
> You listed all methods that may throw an `InvalidStateStoreException`
> atm. For the new exceptions, can any exception be thrown by any method?
> It might help to understand this relationship better.
>
> For example, StreamThreadNotStartedException, seems to only make sense
> for `KafkaStreams#store()`?
>
>
> For `StreamThreadNotRunningException` should we rename it to
> `KafkaStreamsNotRunningException` ?
>
>
> The description of `StreamThreadNotRunningException` and
> `StateStoreNotAvailableException` seems to be the same? From my
> understandng, the description makes sense for
> `StreamThreadNotRunningException` -- for
> `StateStoreNotAvailableException` I was expecting/inferring from the
> name, that it would be thrown if no such store exists in the topology at
> all (ie, user passed in a invalid/wrong store name). For this case, this
> exception should be thrown only from `KafkaStreams#store()` ?
>
>
> For the internal exceptions:
>
> `StateStoreClosedException` -- why can it be wrapped as
> `StreamThreadNotStartedException` ? It seems that the later would only
> be thrown by `KafkaStreams#store()` and thus would be throw directly. A
> closed-exception should only happen after a store was successfully
> retrieved but cannot be queried any longer? Hence, converting/wrapping
> it into a `StateStoreMigratedException` make sense. I am also not sure,
> when a closed-exception would be wrapped by a
> `StateStoreNotAvailableException` (implying my understanding as describe
> above)?
>
> Same questions about `EmptyStateStoreException`.
>
> Thinking about both internal exceptions twice, I am wondering if it
> makes sense to have both internal exceptions at all? I have the
> impression that it make only sense to wrap them with a
> `StateStoreMigragedException`, but if they are wrapped into the same
> exception all the time, we can just remove both and throw
> `StateStoreMigratedException` directly?
>
>
> Last point: Why do we need to add?
>
> > QueryableStoreType#setStreams(KafkaStreams streams);
>
> John asked this question already and you replied to it. But I am not
> sure what your answer means. Can you explain it in more detail?
>
>
>
> Thanks for your patience on this KIP!
>
>
>
> -Matthias
>
>
>
>
>
>
> On 11/11/18 4:55 AM, Vito Jeng wrote:
> > Hi, Matthias,
> >
> > KIP already updated.
> >
> >> - StateStoreClosedException:
> >> will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Can you clarify the cases (ie, when will it be wrapped with the one or
> > the other)?
> >
> > For example, in the implementation(CompositeReadOnlyKeyValueStore#get),
> we
> > get all stores first, and then call ReadOnlyKeyValueStore#get to get
> value
> > in every store iteration.
> >
> > When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
> will
> > be thrown if the state store is not open.
> > We need catch StateStoreClosedException and wrap it in different
> exception
> > type:
> > * If the stream's state is CREATED, we wrap StateStoreClosedException
> > with StreamThreadNotStartedException. User can retry until to RUNNING.
> > * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap StateStoreClosedException with
> > StateStoreMigratedException. User can rediscover the state store.
> > * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap StateStoreClosedException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> >
> >> - StateStoreIsEmptyException:
> >> I don't understand the semantic of this exception. Maybe it's a naming
> > issue?
> >
> > I think yes. :)
> > Does `EmptyStateStoreException` is better ? (already updated in the KIP)
> >
> >
> >> - StateStoreIsEmptyException:
> >> will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Also, can you clarify the cases (ie, when will it be wrapped with the
> one
> > or the other)?
> >
> > For example, in the implementation (CompositeReadOnlyKeyValueStore#get),
> we
> > call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
> > stores. EmptyStateStoreException will be thrown when cannot find any
> store
> > and then we need catch it and wrap it in different exception type:
> > * If the stream's state is CREATED, we wrap EmptyStateStoreException
> with
> > StreamThreadNotStartedException. User can retry until to RUNNING.
> > * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap EmptyStateStoreException with
> > StateStoreMigratedException. User can rediscover the state store.
> > * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap EmptyStateStoreException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> > I hope the above reply can clarify.
> >
> > The last one that was not replied was:
> >
> >> I am also wondering, if we should introduce a fatal exception
> >> `UnkownStateStoreException` to tell users that they passed in an unknown
> >> store name?
> >
> > Until now, unknown state store is not thinking about in the KIP.
> > I believe it would be very useful for users.
> >
> > Looking at the related code(WrappingStoreProvider#stores),
> > I found that I can't distinguish between the state store was migrated or
> an
> > unknown state store.
> >
> > Any thoughts?
> >
> > ---
> > Vito
> >
> >
> >
> > On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw> wrote:
> >
> >> Hi, Matthias,
> >>
> >> Sorry for the late reply.
> >>
> >>> I am wondering what the semantic impact/change is, if we introduce
> >>> `RetryableStateStoreException` and `FatalStateStoreException` that both
> >>> inherit from it. While I like the introduction of both from a high
> level
> >>> point of view, I just want to make sure it's semantically sound and
> >>> backward compatible. Atm, I think it's fine, but I want to point it out
> >>> such that everybody can think about this, too, so we can verify that
> >>> it's a natural evolving API change.
> >>
> >> Thank you for pointing this out. This's really important for public API.
> >>
> >> Just when I was replying to you, I found that KIP needs some modify.
> >> I will fix it ASAP, and then let's continue the discussion.
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
> >> wrote:
> >>
> >>> Hey Vito,
> >>>
> >>> I saw that you updated your PR, but did not reply to my last comments.
> >>> Any thoughts?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
> >>>> Glad to have you back Vito :)
> >>>>
> >>>> Some follow up thoughts:
> >>>>
> >>>> - the current `InvalidStateStoreException` is documents as being
> >>>> sometimes retry-able. From the JavaDocs:
> >>>>
> >>>>> These exceptions may be transient [...] Hence, it is valid to backoff
> >>> and retry when handling this exception.
> >>>>
> >>>> I am wondering what the semantic impact/change is, if we introduce
> >>>> `RetryableStateStoreException` and `FatalStateStoreException` that
> both
> >>>> inherit from it. While I like the introduction of both from a high
> level
> >>>> point of view, I just want to make sure it's semantically sound and
> >>>> backward compatible. Atm, I think it's fine, but I want to point it
> out
> >>>> such that everybody can think about this, too, so we can verify that
> >>>> it's a natural evolving API change.
> >>>>
> >>>> - StateStoreClosedException:
> >>>>
> >>>>> will be wrapped to StateStoreMigratedException or
> >>> StateStoreNotAvailableException later.
> >>>>
> >>>> Can you clarify the cases (ie, when will it be wrapped with the one or
> >>>> the other)?
> >>>>
> >>>> - StateStoreIsEmptyException:
> >>>>
> >>>> I don't understand the semantic of this exception. Maybe it's a naming
> >>>> issue?
> >>>>
> >>>>> will be wrapped to StateStoreMigratedException or
> >>> StateStoreNotAvailableException later.
> >>>>
> >>>> Also, can you clarify the cases (ie, when will it be wrapped with the
> >>>> one or the other)?
> >>>>
> >>>>
> >>>> I am also wondering, if we should introduce a fatal exception
> >>>> `UnkownStateStoreException` to tell users that they passed in an
> unknown
> >>>> store name?
> >>>>
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>>
> >>>> On 10/17/18 8:14 PM, vito jeng wrote:
> >>>>> Just open a PR for further discussion:
> >>>>> https://github.com/apache/kafka/pull/5814
> >>>>>
> >>>>> Any suggestion is welcome.
> >>>>> Thanks!
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>>
> >>>>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
> >>> wrote:
> >>>>>
> >>>>>> Hi John,
> >>>>>>
> >>>>>> Thanks for reviewing the KIP.
> >>>>>>
> >>>>>>> I didn't follow the addition of a new method to the
> >>> QueryableStoreType
> >>>>>>> interface. Can you elaborate why this is necessary to support the
> new
> >>>>>>> exception types?
> >>>>>>
> >>>>>> To support the new exception types, I would check stream state in
> the
> >>>>>> following classes:
> >>>>>> - CompositeReadOnlyKeyValueStore class
> >>>>>> - CompositeReadOnlySessionStore class
> >>>>>> - CompositeReadOnlyWindowStore class
> >>>>>> - DelegatingPeekingKeyValueIterator class
> >>>>>>
> >>>>>> It is also necessary to keep backward compatibility. So I plan
> passing
> >>>>>> stream
> >>>>>> instance to QueryableStoreType instance during KafkaStreams#store()
> >>>>>> invoked.
> >>>>>> It looks a most simple way, I think.
> >>>>>>
> >>>>>> It is why I add a new method to the QueryableStoreType interface. I
> >>> can
> >>>>>> understand
> >>>>>> that we should try to avoid adding the public api method. However,
> at
> >>> the
> >>>>>> moment
> >>>>>> I have no better ideas.
> >>>>>>
> >>>>>> Any thoughts?
> >>>>>>
> >>>>>>
> >>>>>>> Also, looking over your KIP again, it seems valuable to introduce
> >>>>>>> "retriable store exception" and "fatal store exception" marker
> >>> interfaces
> >>>>>>> that the various exceptions can mix in. It would be nice from a
> >>> usability
> >>>>>>> perspective to be able to just log and retry on any "retriable"
> >>> exception
> >>>>>>> and log and shutdown on any fatal exception.
> >>>>>>
> >>>>>> I agree that this is valuable to the user.
> >>>>>> I'll update the KIP.
> >>>>>>
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
> >>> wrote:
> >>>>>>
> >>>>>>> Hi Vito,
> >>>>>>>
> >>>>>>> I'm glad to hear you're well again!
> >>>>>>>
> >>>>>>> I didn't follow the addition of a new method to the
> >>> QueryableStoreType
> >>>>>>> interface. Can you elaborate why this is necessary to support the
> new
> >>>>>>> exception types?
> >>>>>>>
> >>>>>>> Also, looking over your KIP again, it seems valuable to introduce
> >>>>>>> "retriable store exception" and "fatal store exception" marker
> >>> interfaces
> >>>>>>> that the various exceptions can mix in. It would be nice from a
> >>> usability
> >>>>>>> perspective to be able to just log and retry on any "retriable"
> >>> exception
> >>>>>>> and log and shutdown on any fatal exception.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> -John
> >>>>>>>
> >>>>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com>
> >>> wrote:
> >>>>>>>
> >>>>>>>> Thanks for the explanation, that makes sense.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Guozhang
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
> >>> matthias@confluent.io
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> The scenario I had I mind was, that KS is started in one thread
> >>> while
> >>>>>>> a
> >>>>>>>>> second thread has a reference to the object to issue queries.
> >>>>>>>>>
> >>>>>>>>> If a query is issue before the "main thread" started KS, and the
> >>>>>>> "query
> >>>>>>>>> thread" knows that it will eventually get started, it can retry.
> On
> >>>>>>> the
> >>>>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
> >>>>>>> impossible
> >>>>>>>>> to issue any query against it now or in the future and thus the
> >>> error
> >>>>>>> is
> >>>>>>>>> not retryable.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
> >>>>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
> >>>>>>>>>> StreamThreadNotRunning, because I think users' handling logic
> for
> >>>>>>> the
> >>>>>>>>> third
> >>>>>>>>>> case would be likely the same as the second. Do you have some
> >>>>>>> scenarios
> >>>>>>>>>> where users may want to handle them differently?
> >>>>>>>>>>
> >>>>>>>>>> Guozhang
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
> >>>>>>>> matthias@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Sorry to hear! Get well soon!
> >>>>>>>>>>>
> >>>>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free
> to
> >>>>>>> pick
> >>>>>>>> it
> >>>>>>>>>>> up again when you find time.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> >>> error?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
> >>> retryable
> >>>>>>>>> error.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
> not
> >>>>>>>>>>> RUNNING. The
> >>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>>>>
> >>>>>>>>>>> I see. If this is the intention, than I would suggest to have
> two
> >>>>>>> (or
> >>>>>>>>>>> maybe three) different exceptions:
> >>>>>>>>>>>
> >>>>>>>>>>> - StreamThreadRebalancingException (retryable)
> >>>>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
> >>>>>>>>>>> PENDING_SHUTDOWN or DEAD
> >>>>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
> >>>>>>>>>>>
> >>>>>>>>>>> The last one is tricky and could also be merged into one of the
> >>>>>>> first
> >>>>>>>>>>> two, depending if you want to argue that it's retryable or not.
> >>>>>>> (Just
> >>>>>>>>>>> food for though -- not sure what others think.)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -Matthias
> >>>>>>>>>>>
> >>>>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
> >>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thank you for your assistance.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unfortunately, there is no further progress.
> >>>>>>>>>>>> About seven weeks ago, I was injured in sports. I had a broken
> >>>>>>> wrist
> >>>>>>>> on
> >>>>>>>>>>>> my left wrist.
> >>>>>>>>>>>> Many jobs are affected, including this KIP and implementation.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
> Why
> >>>>>>> do
> >>>>>>>> we
> >>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
> >>>>>>> really
> >>>>>>>>> need
> >>>>>>>>>>>>> them? Can't we just throw the correct exception directly
> >>> instead
> >>>>>>> of
> >>>>>>>>>>>>> wrapping it later?
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think you may be right. As I say in the previous:
> >>>>>>>>>>>> "The original idea is that we can distinguish different state
> >>>>>>> store
> >>>>>>>>>>>> exception for different handling. But to be honest, I am not
> >>> quite
> >>>>>>>> sure
> >>>>>>>>>>>> this is necessary. Maybe have some change during
> >>> implementation."
> >>>>>>>>>>>>
> >>>>>>>>>>>> During the implementation, I also feel we maybe not need
> wrapper
> >>>>>>> it.
> >>>>>>>>>>>> We can just throw the correctly directly.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> error?
> >>>>>>>>>>>>
> >>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
> retryable
> >>>>>>>> error.
> >>>>>>>>>>>>
> >>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
> >>>>>>>>> RUNNING.
> >>>>>>>>>>> The
> >>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
> >>> semantics
> >>>>>>> is
> >>>>>>>>>>>> unclear to me atm.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
> >>>>>>>> retryable
> >>>>>>>>>>>> error?
> >>>>>>>>>>>>
> >>>>>>>>>>>> These two comments will be answered in another mail.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
> >>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
> Why
> >>>>>>> do
> >>>>>>>> we
> >>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
> >>>>>>> really
> >>>>>>>>> need
> >>>>>>>>>>>>> them? Can't we just throw the correct exception directly
> >>> instead
> >>>>>>> of
> >>>>>>>>>>>>> wrapping it later?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
> >>> semantics
> >>>>>>> is
> >>>>>>>>>>>>> unclear to me atm.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> error?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
> >>>>>>>> retryable
> >>>>>>>>>>>>> error?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
> >>> key,
> >>>>>>>> long
> >>>>>>>>>>>>> time); that should be added
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Overall I like the KIP but some details are still unclear.
> >>> Maybe
> >>>>>>> it
> >>>>>>>>>>>>> might help if you open an PR in parallel?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
> >>>>>>>>>>>>>> Hi, Guozhang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Bill,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'll try to make some update to make the KIP better.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
> >>> bbejeck@gmail.com
> >>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> At this point, the only thing I would change is possibly
> >>>>>>> removing
> >>>>>>>>> the
> >>>>>>>>>>>>>>> listing of all methods called by the user and the listing
> of
> >>>>>>> all
> >>>>>>>>> store
> >>>>>>>>>>>>>>> types and focus on what states result in which exceptions
> >>>>>>> thrown
> >>>>>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Bill
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
> >>>>>>>> wangguoz@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me. I'm
> +1
> >>>>>>> on
> >>>>>>>> the
> >>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> About the base class InvalidStateStoreException itself,
> I'd
> >>>>>>>>> actually
> >>>>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as part
> >>> of
> >>>>>>> the
> >>>>>>>>>>>>> public
> >>>>>>>>>>>>>>>> API, for people who do not want to handle these cases
> >>>>>>> differently
> >>>>>>>>> (if
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all
> three
> >>>>>>>>>>> exceptions
> >>>>>>>>>>>>>>>> one-by-one).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
> >>>>>>> john@confluent.io
> >>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think it's much nicer to give callers different
> >>> exceptions
> >>>>>>> to
> >>>>>>>>> tell
> >>>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>> whether the state store got migrated, whether it's still
> >>>>>>>>>>> initializing,
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> whether there's some unrecoverable error.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
> >>>>>>>>> non-user-facing
> >>>>>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>> such as what exceptions we will throw internally. The KIP
> >>> is
> >>>>>>>>>>> primarily
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> discuss public interface changes.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> You might consider simply removing all the internal
> details
> >>>>>>> from
> >>>>>>>>> the
> >>>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
> >>>>>>> smaller
> >>>>>>>>> and
> >>>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
> >>>>>>> internal
> >>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>> when it comes to implementation.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I like your decision to have your refined exceptions
> extend
> >>>>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
> >>> compatibility.
> >>>>>>>> Since
> >>>>>>>>>>> we
> >>>>>>>>>>>>>>>> want
> >>>>>>>>>>>>>>>>> to encourage callers to catch the more specific
> exceptions,
> >>>>>>> and
> >>>>>>>> we
> >>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
> >>>>>>> anymore,
> >>>>>>>> you
> >>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
> >>>>>>>>>>>>>>> InvalidStateStoreException.
> >>>>>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
> >>>>>>>> exception
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>> open
> >>>>>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
> >>>>>>> entirely
> >>>>>>>>> in a
> >>>>>>>>>>>>>>>> future
> >>>>>>>>>>>>>>>>> release.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions
> explicit?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for the feedback!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
> >>> not.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Got it!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
> used?
> >>> Do
> >>>>>>>> you
> >>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> >>>>>>>> anymore,
> >>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
> >>> understanding).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
> >>>>>>> thrown
> >>>>>>>>> at
> >>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>> anymore,
> >>>>>>>>>>>>>>>>>>> but only new sub-classes.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
> >>>>>>> exception
> >>>>>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
> >>> will
> >>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
> >>>>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
> >>>>>>>>>>>>>>>>>>> `StateStoreRetryableException` or
> >>> `StateStoreFailException`
> >>>>>>>>> during
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
> >>> exceptions".
> >>>>>>>> It's
> >>>>>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
> >>>>>>>>> distinguish
> >>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
> >>>>>>>>>>>>>>>>>>> The original idea is that we can distinguish different
> >>>>>>> state
> >>>>>>>>> store
> >>>>>>>>>>>>>>>>>>> exception for
> >>>>>>>>>>>>>>>>>>> different handling.
> >>>>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is
> necessary.
> >>>>>>> Maybe
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> some change during implementation.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Does it make sense?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the update Vito!
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
> >>> not.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
> used?
> >>> Do
> >>>>>>>> you
> >>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> >>>>>>>> anymore,
> >>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
> >>> understanding).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
> >>>>>>> exception
> >>>>>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
> >>> exceptions".
> >>>>>>>> It's
> >>>>>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the review.
> >>>>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
> >>>>>>> pause...
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
> >>> code.
> >>>>>>>> It's
> >>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are affected.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
> >>>>>>> `Changes
> >>>>>>>> in
> >>>>>>>>>>>>>>> call
> >>>>>>>>>>>>>>>>>>>> trace`.
> >>>>>>>>>>>>>>>>>>>>> Do you think it should be removed?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP,
> the
> >>>>>>> main
> >>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
> >>>>>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
> >>> things.
> >>>>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions
> extend
> >>>>>>> from
> >>>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Some questions:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
> >>>>>>> `InvalidStateStoreException`
> >>>>>>>> be
> >>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> this purpose?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Does this question miss some word?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - What the superclass of
> >>>>>>> `StateStoreStreamThreadNotRunni
> >>>>>>>>>>>>>>>>> ngException`
> >>>>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
> >>> exception
> >>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
> >>> modify
> >>>>>>>> some
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
> >>>>>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the
> state
> >>>>>>> both
> >>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams
> thread
> >>>>>>> and
> >>>>>>>>> state
> >>>>>>>>>>>>>>>>> store,
> >>>>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax
> <
> >>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is
> >>> not
> >>>>>>>>> hosted
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
> >>>>>>> state
> >>>>>>>>>>>>>>>> rebalance
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
> >>>>>>> store.
> >>>>>>>>> For
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> second
> >>>>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
> >>> finished.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
> >>>>>>> PENDING_SHUTDOWN, or
> >>>>>>>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
> >>>>>>>> rediscover
> >>>>>>>>> or
> >>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
> >>>>>>> user
> >>>>>>>> can
> >>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>> plain
> >>>>>>>>>>>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN
> >>> or
> >>>>>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the
> >>> user
> >>>>>>>> needs
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> rediscover
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J.
> Sax
> >>> <
> >>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
> >>>>>>> idea of
> >>>>>>>>> the
> >>>>>>>>>>>>>>>> Jira
> >>>>>>>>>>>>>>>>>>>> was,
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
> >>> "recovery"
> >>>>>>>>>>>>>>>> strategies
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently
> recreated,
> >>> a
> >>>>>>>> user
> >>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> wait
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other
> >>> hand,
> >>>>>>> if
> >>>>>>>> a
> >>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>>>>>>> migrated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to
> rediscover
> >>>>>>> the
> >>>>>>>>> store
> >>>>>>>>>>>>>>>>>> instead
> >>>>>>>>>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
> >>>>>>> talks
> >>>>>>>>> about
> >>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> ideas
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
> >>> figure
> >>>>>>>> out
> >>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
> >>>>>>> users, if
> >>>>>>>>> we
> >>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>> wants
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then
> probably
> >>> we
> >>>>>>>>> should
> >>>>>>>>>>>>>>>>> rename
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>>>>>>>>>>>>>>> StateStoreClosedException
> >>>>>>>>>>>>>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are
> the
> >>>>>>>>> possible
> >>>>>>>>>>>>>>>>>> internal
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the
> >>> call
> >>>>>>>>> trace,
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others,
> >>> and
> >>>>>>>> which
> >>>>>>>>>>>>>>> ones
> >>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
> >>>>>>> should
> >>>>>>>> not
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> wrapped
> >>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> >>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
> >>>>>>>>> confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>>>>>>>>>>>>>>> different+errors
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> -- Guozhang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> -- Guozhang
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, Mattias,
Just completed the modification of KIP, please take a look when you are
available.
---
Vito
On Wed, Jul 3, 2019 at 9:07 PM Vito Jeng <vi...@is-land.com.tw> wrote:
> Hi, Matthias,
>
> This is second part.
>
> > For the internal exceptions:
> >
> > `StateStoreClosedException` -- why can it be wrapped as
> > `StreamThreadNotStartedException` ? It seems that the later would only
> > be thrown by `KafkaStreams#store()` and thus would be throw directly.
>
> Both `StateStoreClosedException` and `EmptyStateStoreException` not can be
> wrapped as `StreamThreadNotStartedException`.
> This is a mistaken written in the previous KIP. Thank you point this.
>
> > A closed-exception should only happen after a store was successfully
> > retrieved but cannot be queried any longer? Hence, converting/wrapping
> > it into a `StateStoreMigratedException` make sense. I am also not sure,
> > when a closed-exception would be wrapped by a
> > `StateStoreNotAvailableException` (implying my understanding as describe
> > above)?
> >
> > Same questions about `EmptyStateStoreException`.
> >
> > Thinking about both internal exceptions twice, I am wondering if it
> > makes sense to have both internal exceptions at all? I have the
> > impression that it make only sense to wrap them with a
> > `StateStoreMigragedException`, but if they are wrapped into the same
> > exception all the time, we can just remove both and throw
> > `StateStoreMigratedException` directly?
>
> After deeper thinking, I think you are right. It seems we can throw
> `StateStoreMigratedException` directly.
> So that we can remove `StateStoreClosedException`,
> `EmptyStateStoreException` and `StateStoreNotAvailableException`.
> Will update the KIP.
>
> BTW, if we remove above three exceptions, the
> `StreamThreadNotRunningException` will be the only one sub class extends
> from FatalStateStoreException.
> Should we remove `StreamThreadNotRunningException` and throw
> `FatalStateStoreException` directly ?
>
> > Last point: Why do we need to add?
> > QueryableStoreType#setStreams(KafkaStreams streams);
> > John asked this question already and you replied to it. But I am not
> > sure what your answer means. Can you explain it in more detail?
>
> The main purpose is to pass the KafkaStreams reference into
> CompositeReadOnlyKeyValueStore / CompositeReadOnlySessionStore/
> CompositeReadOnlyWindowStore instance.
> We need check KafkaStreams state to warp InvalidStateStoreException in to
> other exception(e.g., StateStoreMigratedException) when the user accesses
> these read-only stores.
>
> The original thought is to add `setStreams` method in to
> QueryableStoreType. But now I think I find a better way during recent days.
> This way does not need to change any public interface. So we can skip this
> question. :)
>
>
> I will update the KIP based on our discussion.
> Thank you for help to finish the KIP!
>
> ---
> Vito
>
>
> On Thu, Jun 6, 2019 at 8:23 AM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Hi Vito,
>>
>> sorry for dropping this discussion on the floor a while back. I was just
>> re-reading the KIP and discussion thread, and I think it is shaping out
>> nicely!
>>
>> I like the overall hierarchy of the exception classes.
>>
>> Some things are still not 100% clear:
>>
>>
>> You listed all methods that may throw an `InvalidStateStoreException`
>> atm. For the new exceptions, can any exception be thrown by any method?
>> It might help to understand this relationship better.
>>
>> For example, StreamThreadNotStartedException, seems to only make sense
>> for `KafkaStreams#store()`?
>>
>>
>> For `StreamThreadNotRunningException` should we rename it to
>> `KafkaStreamsNotRunningException` ?
>>
>>
>> The description of `StreamThreadNotRunningException` and
>> `StateStoreNotAvailableException` seems to be the same? From my
>> understandng, the description makes sense for
>> `StreamThreadNotRunningException` -- for
>> `StateStoreNotAvailableException` I was expecting/inferring from the
>> name, that it would be thrown if no such store exists in the topology at
>> all (ie, user passed in a invalid/wrong store name). For this case, this
>> exception should be thrown only from `KafkaStreams#store()` ?
>>
>>
>> For the internal exceptions:
>>
>> `StateStoreClosedException` -- why can it be wrapped as
>> `StreamThreadNotStartedException` ? It seems that the later would only
>> be thrown by `KafkaStreams#store()` and thus would be throw directly. A
>> closed-exception should only happen after a store was successfully
>> retrieved but cannot be queried any longer? Hence, converting/wrapping
>> it into a `StateStoreMigratedException` make sense. I am also not sure,
>> when a closed-exception would be wrapped by a
>> `StateStoreNotAvailableException` (implying my understanding as describe
>> above)?
>>
>> Same questions about `EmptyStateStoreException`.
>>
>> Thinking about both internal exceptions twice, I am wondering if it
>> makes sense to have both internal exceptions at all? I have the
>> impression that it make only sense to wrap them with a
>> `StateStoreMigragedException`, but if they are wrapped into the same
>> exception all the time, we can just remove both and throw
>> `StateStoreMigratedException` directly?
>>
>>
>> Last point: Why do we need to add?
>>
>> > QueryableStoreType#setStreams(KafkaStreams streams);
>>
>> John asked this question already and you replied to it. But I am not
>> sure what your answer means. Can you explain it in more detail?
>>
>>
>>
>> Thanks for your patience on this KIP!
>>
>>
>>
>> -Matthias
>>
>>
>>
>>
>>
>>
>> On 11/11/18 4:55 AM, Vito Jeng wrote:
>> > Hi, Matthias,
>> >
>> > KIP already updated.
>> >
>> >> - StateStoreClosedException:
>> >> will be wrapped to StateStoreMigratedException or
>> > StateStoreNotAvailableException later.
>> >> Can you clarify the cases (ie, when will it be wrapped with the one or
>> > the other)?
>> >
>> > For example, in the implementation(CompositeReadOnlyKeyValueStore#get),
>> we
>> > get all stores first, and then call ReadOnlyKeyValueStore#get to get
>> value
>> > in every store iteration.
>> >
>> > When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
>> will
>> > be thrown if the state store is not open.
>> > We need catch StateStoreClosedException and wrap it in different
>> exception
>> > type:
>> > * If the stream's state is CREATED, we wrap StateStoreClosedException
>> > with StreamThreadNotStartedException. User can retry until to RUNNING.
>> > * If the stream's state is RUNNING / REBALANCING, the state store
>> should
>> > be migrated, we wrap StateStoreClosedException with
>> > StateStoreMigratedException. User can rediscover the state store.
>> > * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
>> > stream thread is not available, we wrap StateStoreClosedException with
>> > StateStoreNotAvailableException. User cannot retry when this exception
>> is
>> > thrown.
>> >
>> >
>> >> - StateStoreIsEmptyException:
>> >> I don't understand the semantic of this exception. Maybe it's a naming
>> > issue?
>> >
>> > I think yes. :)
>> > Does `EmptyStateStoreException` is better ? (already updated in the KIP)
>> >
>> >
>> >> - StateStoreIsEmptyException:
>> >> will be wrapped to StateStoreMigratedException or
>> > StateStoreNotAvailableException later.
>> >> Also, can you clarify the cases (ie, when will it be wrapped with the
>> one
>> > or the other)?
>> >
>> > For example, in the implementation
>> (CompositeReadOnlyKeyValueStore#get), we
>> > call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
>> > stores. EmptyStateStoreException will be thrown when cannot find any
>> store
>> > and then we need catch it and wrap it in different exception type:
>> > * If the stream's state is CREATED, we wrap EmptyStateStoreException
>> with
>> > StreamThreadNotStartedException. User can retry until to RUNNING.
>> > * If the stream's state is RUNNING / REBALANCING, the state store
>> should
>> > be migrated, we wrap EmptyStateStoreException with
>> > StateStoreMigratedException. User can rediscover the state store.
>> > * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
>> > stream thread is not available, we wrap EmptyStateStoreException with
>> > StateStoreNotAvailableException. User cannot retry when this exception
>> is
>> > thrown.
>> >
>> > I hope the above reply can clarify.
>> >
>> > The last one that was not replied was:
>> >
>> >> I am also wondering, if we should introduce a fatal exception
>> >> `UnkownStateStoreException` to tell users that they passed in an
>> unknown
>> >> store name?
>> >
>> > Until now, unknown state store is not thinking about in the KIP.
>> > I believe it would be very useful for users.
>> >
>> > Looking at the related code(WrappingStoreProvider#stores),
>> > I found that I can't distinguish between the state store was migrated
>> or an
>> > unknown state store.
>> >
>> > Any thoughts?
>> >
>> > ---
>> > Vito
>> >
>> >
>> >
>> > On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw> wrote:
>> >
>> >> Hi, Matthias,
>> >>
>> >> Sorry for the late reply.
>> >>
>> >>> I am wondering what the semantic impact/change is, if we introduce
>> >>> `RetryableStateStoreException` and `FatalStateStoreException` that
>> both
>> >>> inherit from it. While I like the introduction of both from a high
>> level
>> >>> point of view, I just want to make sure it's semantically sound and
>> >>> backward compatible. Atm, I think it's fine, but I want to point it
>> out
>> >>> such that everybody can think about this, too, so we can verify that
>> >>> it's a natural evolving API change.
>> >>
>> >> Thank you for pointing this out. This's really important for public
>> API.
>> >>
>> >> Just when I was replying to you, I found that KIP needs some modify.
>> >> I will fix it ASAP, and then let's continue the discussion.
>> >>
>> >> ---
>> >> Vito
>> >>
>> >>
>> >> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
>> >> wrote:
>> >>
>> >>> Hey Vito,
>> >>>
>> >>> I saw that you updated your PR, but did not reply to my last comments.
>> >>> Any thoughts?
>> >>>
>> >>>
>> >>> -Matthias
>> >>>
>> >>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
>> >>>> Glad to have you back Vito :)
>> >>>>
>> >>>> Some follow up thoughts:
>> >>>>
>> >>>> - the current `InvalidStateStoreException` is documents as being
>> >>>> sometimes retry-able. From the JavaDocs:
>> >>>>
>> >>>>> These exceptions may be transient [...] Hence, it is valid to
>> backoff
>> >>> and retry when handling this exception.
>> >>>>
>> >>>> I am wondering what the semantic impact/change is, if we introduce
>> >>>> `RetryableStateStoreException` and `FatalStateStoreException` that
>> both
>> >>>> inherit from it. While I like the introduction of both from a high
>> level
>> >>>> point of view, I just want to make sure it's semantically sound and
>> >>>> backward compatible. Atm, I think it's fine, but I want to point it
>> out
>> >>>> such that everybody can think about this, too, so we can verify that
>> >>>> it's a natural evolving API change.
>> >>>>
>> >>>> - StateStoreClosedException:
>> >>>>
>> >>>>> will be wrapped to StateStoreMigratedException or
>> >>> StateStoreNotAvailableException later.
>> >>>>
>> >>>> Can you clarify the cases (ie, when will it be wrapped with the one
>> or
>> >>>> the other)?
>> >>>>
>> >>>> - StateStoreIsEmptyException:
>> >>>>
>> >>>> I don't understand the semantic of this exception. Maybe it's a
>> naming
>> >>>> issue?
>> >>>>
>> >>>>> will be wrapped to StateStoreMigratedException or
>> >>> StateStoreNotAvailableException later.
>> >>>>
>> >>>> Also, can you clarify the cases (ie, when will it be wrapped with the
>> >>>> one or the other)?
>> >>>>
>> >>>>
>> >>>> I am also wondering, if we should introduce a fatal exception
>> >>>> `UnkownStateStoreException` to tell users that they passed in an
>> unknown
>> >>>> store name?
>> >>>>
>> >>>>
>> >>>>
>> >>>> -Matthias
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 10/17/18 8:14 PM, vito jeng wrote:
>> >>>>> Just open a PR for further discussion:
>> >>>>> https://github.com/apache/kafka/pull/5814
>> >>>>>
>> >>>>> Any suggestion is welcome.
>> >>>>> Thanks!
>> >>>>>
>> >>>>> ---
>> >>>>> Vito
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
>> >>> wrote:
>> >>>>>
>> >>>>>> Hi John,
>> >>>>>>
>> >>>>>> Thanks for reviewing the KIP.
>> >>>>>>
>> >>>>>>> I didn't follow the addition of a new method to the
>> >>> QueryableStoreType
>> >>>>>>> interface. Can you elaborate why this is necessary to support the
>> new
>> >>>>>>> exception types?
>> >>>>>>
>> >>>>>> To support the new exception types, I would check stream state in
>> the
>> >>>>>> following classes:
>> >>>>>> - CompositeReadOnlyKeyValueStore class
>> >>>>>> - CompositeReadOnlySessionStore class
>> >>>>>> - CompositeReadOnlyWindowStore class
>> >>>>>> - DelegatingPeekingKeyValueIterator class
>> >>>>>>
>> >>>>>> It is also necessary to keep backward compatibility. So I plan
>> passing
>> >>>>>> stream
>> >>>>>> instance to QueryableStoreType instance during KafkaStreams#store()
>> >>>>>> invoked.
>> >>>>>> It looks a most simple way, I think.
>> >>>>>>
>> >>>>>> It is why I add a new method to the QueryableStoreType interface. I
>> >>> can
>> >>>>>> understand
>> >>>>>> that we should try to avoid adding the public api method. However,
>> at
>> >>> the
>> >>>>>> moment
>> >>>>>> I have no better ideas.
>> >>>>>>
>> >>>>>> Any thoughts?
>> >>>>>>
>> >>>>>>
>> >>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>> >>>>>>> "retriable store exception" and "fatal store exception" marker
>> >>> interfaces
>> >>>>>>> that the various exceptions can mix in. It would be nice from a
>> >>> usability
>> >>>>>>> perspective to be able to just log and retry on any "retriable"
>> >>> exception
>> >>>>>>> and log and shutdown on any fatal exception.
>> >>>>>>
>> >>>>>> I agree that this is valuable to the user.
>> >>>>>> I'll update the KIP.
>> >>>>>>
>> >>>>>>
>> >>>>>> Thanks
>> >>>>>>
>> >>>>>>
>> >>>>>> ---
>> >>>>>> Vito
>> >>>>>>
>> >>>>>>
>> >>>>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
>> >>> wrote:
>> >>>>>>
>> >>>>>>> Hi Vito,
>> >>>>>>>
>> >>>>>>> I'm glad to hear you're well again!
>> >>>>>>>
>> >>>>>>> I didn't follow the addition of a new method to the
>> >>> QueryableStoreType
>> >>>>>>> interface. Can you elaborate why this is necessary to support the
>> new
>> >>>>>>> exception types?
>> >>>>>>>
>> >>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>> >>>>>>> "retriable store exception" and "fatal store exception" marker
>> >>> interfaces
>> >>>>>>> that the various exceptions can mix in. It would be nice from a
>> >>> usability
>> >>>>>>> perspective to be able to just log and retry on any "retriable"
>> >>> exception
>> >>>>>>> and log and shutdown on any fatal exception.
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> -John
>> >>>>>>>
>> >>>>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wangguoz@gmail.com
>> >
>> >>> wrote:
>> >>>>>>>
>> >>>>>>>> Thanks for the explanation, that makes sense.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Guozhang
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
>> >>> matthias@confluent.io
>> >>>>>>>>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> The scenario I had I mind was, that KS is started in one thread
>> >>> while
>> >>>>>>> a
>> >>>>>>>>> second thread has a reference to the object to issue queries.
>> >>>>>>>>>
>> >>>>>>>>> If a query is issue before the "main thread" started KS, and the
>> >>>>>>> "query
>> >>>>>>>>> thread" knows that it will eventually get started, it can
>> retry. On
>> >>>>>>> the
>> >>>>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>> >>>>>>> impossible
>> >>>>>>>>> to issue any query against it now or in the future and thus the
>> >>> error
>> >>>>>>> is
>> >>>>>>>>> not retryable.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> -Matthias
>> >>>>>>>>>
>> >>>>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
>> >>>>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
>> >>>>>>>>>> StreamThreadNotRunning, because I think users' handling logic
>> for
>> >>>>>>> the
>> >>>>>>>>> third
>> >>>>>>>>>> case would be likely the same as the second. Do you have some
>> >>>>>>> scenarios
>> >>>>>>>>>> where users may want to handle them differently?
>> >>>>>>>>>>
>> >>>>>>>>>> Guozhang
>> >>>>>>>>>>
>> >>>>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>> >>>>>>>> matthias@confluent.io>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Sorry to hear! Get well soon!
>> >>>>>>>>>>>
>> >>>>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free
>> to
>> >>>>>>> pick
>> >>>>>>>> it
>> >>>>>>>>>>> up again when you find time.
>> >>>>>>>>>>>
>> >>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>> >>> error?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>> >>> retryable
>> >>>>>>>>> error.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>> >>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
>> not
>> >>>>>>>>>>> RUNNING. The
>> >>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I see. If this is the intention, than I would suggest to have
>> two
>> >>>>>>> (or
>> >>>>>>>>>>> maybe three) different exceptions:
>> >>>>>>>>>>>
>> >>>>>>>>>>> - StreamThreadRebalancingException (retryable)
>> >>>>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
>> >>>>>>>>>>> PENDING_SHUTDOWN or DEAD
>> >>>>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
>> >>>>>>>>>>>
>> >>>>>>>>>>> The last one is tricky and could also be merged into one of
>> the
>> >>>>>>> first
>> >>>>>>>>>>> two, depending if you want to argue that it's retryable or
>> not.
>> >>>>>>> (Just
>> >>>>>>>>>>> food for though -- not sure what others think.)
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> -Matthias
>> >>>>>>>>>>>
>> >>>>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
>> >>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thank you for your assistance.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> what is the status of this KIP?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Unfortunately, there is no further progress.
>> >>>>>>>>>>>> About seven weeks ago, I was injured in sports. I had a
>> broken
>> >>>>>>> wrist
>> >>>>>>>> on
>> >>>>>>>>>>>> my left wrist.
>> >>>>>>>>>>>> Many jobs are affected, including this KIP and
>> implementation.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
>> Why
>> >>>>>>> do
>> >>>>>>>> we
>> >>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>> >>>>>>> really
>> >>>>>>>>> need
>> >>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>> >>> instead
>> >>>>>>> of
>> >>>>>>>>>>>>> wrapping it later?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I think you may be right. As I say in the previous:
>> >>>>>>>>>>>> "The original idea is that we can distinguish different state
>> >>>>>>> store
>> >>>>>>>>>>>> exception for different handling. But to be honest, I am not
>> >>> quite
>> >>>>>>>> sure
>> >>>>>>>>>>>> this is necessary. Maybe have some change during
>> >>> implementation."
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> During the implementation, I also feel we maybe not need
>> wrapper
>> >>>>>>> it.
>> >>>>>>>>>>>> We can just throw the correctly directly.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>> error?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>> retryable
>> >>>>>>>> error.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>> >>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
>> not
>> >>>>>>>>> RUNNING.
>> >>>>>>>>>>> The
>> >>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>> >>> semantics
>> >>>>>>> is
>> >>>>>>>>>>>> unclear to me atm.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>> >>>>>>>> retryable
>> >>>>>>>>>>>> error?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> These two comments will be answered in another mail.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> ---
>> >>>>>>>>>>>> Vito
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>> >>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Vito,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> what is the status of this KIP?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
>> Why
>> >>>>>>> do
>> >>>>>>>> we
>> >>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>> >>>>>>> really
>> >>>>>>>>> need
>> >>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>> >>> instead
>> >>>>>>> of
>> >>>>>>>>>>>>> wrapping it later?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>> >>> semantics
>> >>>>>>> is
>> >>>>>>>>>>>>> unclear to me atm.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>> error?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>> >>>>>>>> retryable
>> >>>>>>>>>>>>> error?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
>> >>> key,
>> >>>>>>>> long
>> >>>>>>>>>>>>> time); that should be added
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Overall I like the KIP but some details are still unclear.
>> >>> Maybe
>> >>>>>>> it
>> >>>>>>>>>>>>> might help if you open an PR in parallel?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>> >>>>>>>>>>>>>> Hi, Guozhang,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks for the comment!
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Hi, Bill,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I'll try to make some update to make the KIP better.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks for the comment!
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
>> >>> bbejeck@gmail.com
>> >>>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hi Vito,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> At this point, the only thing I would change is possibly
>> >>>>>>> removing
>> >>>>>>>>> the
>> >>>>>>>>>>>>>>> listing of all methods called by the user and the listing
>> of
>> >>>>>>> all
>> >>>>>>>>> store
>> >>>>>>>>>>>>>>> types and focus on what states result in which exceptions
>> >>>>>>> thrown
>> >>>>>>>> to
>> >>>>>>>>>>> the
>> >>>>>>>>>>>>>>> user.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>> Bill
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>> >>>>>>>> wangguoz@gmail.com>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me.
>> I'm +1
>> >>>>>>> on
>> >>>>>>>> the
>> >>>>>>>>>>>>> KIP.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> About the base class InvalidStateStoreException itself,
>> I'd
>> >>>>>>>>> actually
>> >>>>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as
>> part
>> >>> of
>> >>>>>>> the
>> >>>>>>>>>>>>> public
>> >>>>>>>>>>>>>>>> API, for people who do not want to handle these cases
>> >>>>>>> differently
>> >>>>>>>>> (if
>> >>>>>>>>>>>>> we
>> >>>>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all
>> three
>> >>>>>>>>>>> exceptions
>> >>>>>>>>>>>>>>>> one-by-one).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Guozhang
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>> >>>>>>> john@confluent.io
>> >>>>>>>>>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Hi Vito,
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Thanks for the KIP!
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I think it's much nicer to give callers different
>> >>> exceptions
>> >>>>>>> to
>> >>>>>>>>> tell
>> >>>>>>>>>>>>>>> them
>> >>>>>>>>>>>>>>>>> whether the state store got migrated, whether it's still
>> >>>>>>>>>>> initializing,
>> >>>>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>> whether there's some unrecoverable error.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
>> >>>>>>>>> non-user-facing
>> >>>>>>>>>>>>>>>> details
>> >>>>>>>>>>>>>>>>> such as what exceptions we will throw internally. The
>> KIP
>> >>> is
>> >>>>>>>>>>> primarily
>> >>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>> discuss public interface changes.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> You might consider simply removing all the internal
>> details
>> >>>>>>> from
>> >>>>>>>>> the
>> >>>>>>>>>>>>>>> KIP,
>> >>>>>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
>> >>>>>>> smaller
>> >>>>>>>>> and
>> >>>>>>>>>>>>>>>> easier
>> >>>>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
>> >>>>>>> internal
>> >>>>>>>>>>>>> details
>> >>>>>>>>>>>>>>>>> when it comes to implementation.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I like your decision to have your refined exceptions
>> extend
>> >>>>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
>> >>> compatibility.
>> >>>>>>>> Since
>> >>>>>>>>>>> we
>> >>>>>>>>>>>>>>>> want
>> >>>>>>>>>>>>>>>>> to encourage callers to catch the more specific
>> exceptions,
>> >>>>>>> and
>> >>>>>>>> we
>> >>>>>>>>>>>>>>> don't
>> >>>>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
>> >>>>>>> anymore,
>> >>>>>>>> you
>> >>>>>>>>>>>>>>> might
>> >>>>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
>> >>>>>>>>>>>>>>> InvalidStateStoreException.
>> >>>>>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
>> >>>>>>>> exception
>> >>>>>>>>>>> and
>> >>>>>>>>>>>>>>>> open
>> >>>>>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
>> >>>>>>> entirely
>> >>>>>>>>> in a
>> >>>>>>>>>>>>>>>> future
>> >>>>>>>>>>>>>>>>> release.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>>>> -John
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>> >>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions
>> explicit?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Thanks for the feedback!
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>> >>> not.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Got it!
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>> >>>>>>>>>>>>>>> `StateStoreFailException`
>> >>>>>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
>> used?
>> >>> Do
>> >>>>>>>> you
>> >>>>>>>>>>>>>>>> suggest
>> >>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
>> all
>> >>>>>>>> anymore,
>> >>>>>>>>>>>>>>> but
>> >>>>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>> >>> understanding).
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is
>> not
>> >>>>>>> thrown
>> >>>>>>>>> at
>> >>>>>>>>>>>>>>> all
>> >>>>>>>>>>>>>>>>>>> anymore,
>> >>>>>>>>>>>>>>>>>>> but only new sub-classes.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>> >>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>> >>>>>>> exception
>> >>>>>>>>>>>>>>>> finally.
>> >>>>>>>>>>>>>>>>>>>> Can you elaborate?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
>> >>> will
>> >>>>>>>>> throw
>> >>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>> >>>>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
>> >>>>>>>>>>>>>>>>>>> `StateStoreRetryableException` or
>> >>> `StateStoreFailException`
>> >>>>>>>>> during
>> >>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>> >>> exceptions".
>> >>>>>>>> It's
>> >>>>>>>>>>>>>>>> unclear
>> >>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>> >>>>>>>>> distinguish
>> >>>>>>>>>>>>>>>>> between
>> >>>>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
>> >>>>>>>>>>>>>>>>>>> The original idea is that we can distinguish different
>> >>>>>>> state
>> >>>>>>>>> store
>> >>>>>>>>>>>>>>>>>>> exception for
>> >>>>>>>>>>>>>>>>>>> different handling.
>> >>>>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is
>> necessary.
>> >>>>>>> Maybe
>> >>>>>>>>>>> have
>> >>>>>>>>>>>>>>>>>>> some change during implementation.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Does it make sense?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>> >>>>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Thanks for the update Vito!
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>> >>> not.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>> >>>>>>>>>>>>>>> `StateStoreFailException`
>> >>>>>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
>> used?
>> >>> Do
>> >>>>>>>> you
>> >>>>>>>>>>>>>>>> suggest
>> >>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at
>> all
>> >>>>>>>> anymore,
>> >>>>>>>>>>>>>>> but
>> >>>>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>> >>> understanding).
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>> >>>>>>> exception
>> >>>>>>>>>>>>>>>> finally.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Can you elaborate?
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>> >>> exceptions".
>> >>>>>>>> It's
>> >>>>>>>>>>>>>>>> unclear
>> >>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Thanks for the review.
>> >>>>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>> >>>>>>>>>>>>>>>>> matthias@confluent.io
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>> >>>>>>> pause...
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
>> >>> code.
>> >>>>>>>> It's
>> >>>>>>>>>>>>>>>> useful
>> >>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are
>> affected.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
>> >>>>>>> `Changes
>> >>>>>>>> in
>> >>>>>>>>>>>>>>> call
>> >>>>>>>>>>>>>>>>>>>> trace`.
>> >>>>>>>>>>>>>>>>>>>>> Do you think it should be removed?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP,
>> the
>> >>>>>>> main
>> >>>>>>>>>>>>>>> change
>> >>>>>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
>> >>>>>>>>>>>>>>> `InvalidStateStoreException`.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
>> >>> things.
>> >>>>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions
>> extend
>> >>>>>>> from
>> >>>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Some questions:
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
>> >>>>>>> `InvalidStateStoreException`
>> >>>>>>>> be
>> >>>>>>>>>>>>>>> used
>> >>>>>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>>>>>>> this purpose?
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Does this question miss some word?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> - What the superclass of
>> >>>>>>> `StateStoreStreamThreadNotRunni
>> >>>>>>>>>>>>>>>>> ngException`
>> >>>>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>> >>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>> >>>>>>>>>>>>>>>>>>>>>> ?
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
>> >>> exception
>> >>>>>>> ?
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
>> >>> modify
>> >>>>>>>> some
>> >>>>>>>>>>>>>>> code
>> >>>>>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
>> >>>>>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
>> >>>>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>> >>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the
>> state
>> >>>>>>> both
>> >>>>>>>>> of
>> >>>>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams
>> thread
>> >>>>>>> and
>> >>>>>>>>> state
>> >>>>>>>>>>>>>>>>> store,
>> >>>>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J.
>> Sax <
>> >>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Vito,
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is
>> >>> not
>> >>>>>>>>> hosted
>> >>>>>>>>>>>>>>> an
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
>> >>>>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>> >>>>>>> state
>> >>>>>>>>>>>>>>>> rebalance
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover
>> the
>> >>>>>>> store.
>> >>>>>>>>> For
>> >>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>> second
>> >>>>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
>> >>> finished.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>> >>>>>>> PENDING_SHUTDOWN, or
>> >>>>>>>>>>>>>>>>>> NOT_RUNNING,
>> >>>>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>> >>>>>>>> rediscover
>> >>>>>>>>> or
>> >>>>>>>>>>>>>>>>> retry.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means
>> the
>> >>>>>>> user
>> >>>>>>>> can
>> >>>>>>>>>>>>>>> just
>> >>>>>>>>>>>>>>>>>> plain
>> >>>>>>>>>>>>>>>>>>>>>>>>>> retry.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or
>> PENDING_SHUTDOWN
>> >>> or
>> >>>>>>>>>>>>>>>> NOT_RUNNING,
>> >>>>>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>>>>>>>>> means
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the
>> >>> user
>> >>>>>>>> needs
>> >>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>> rediscover
>> >>>>>>>>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> store.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J.
>> Sax
>> >>> <
>> >>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>> >>>>>>> idea of
>> >>>>>>>>> the
>> >>>>>>>>>>>>>>>> Jira
>> >>>>>>>>>>>>>>>>>>>> was,
>> >>>>>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
>> >>> "recovery"
>> >>>>>>>>>>>>>>>> strategies
>> >>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>>>> user.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently
>> recreated,
>> >>> a
>> >>>>>>>> user
>> >>>>>>>>>>>>>>> just
>> >>>>>>>>>>>>>>>>> need
>> >>>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>>>> wait
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other
>> >>> hand,
>> >>>>>>> if
>> >>>>>>>> a
>> >>>>>>>>>>>>>>> store
>> >>>>>>>>>>>>>>>>> go
>> >>>>>>>>>>>>>>>>>>>>>>>> migrated
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to
>> rediscover
>> >>>>>>> the
>> >>>>>>>>> store
>> >>>>>>>>>>>>>>>>>> instead
>> >>>>>>>>>>>>>>>>>>>>>> of a
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>> >>>>>>> talks
>> >>>>>>>>> about
>> >>>>>>>>>>>>>>>> this
>> >>>>>>>>>>>>>>>>>>>> ideas
>> >>>>>>>>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
>> >>> figure
>> >>>>>>>> out
>> >>>>>>>>>>>>>>> first
>> >>>>>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>>>>>>> what
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>> >>>>>>> users, if
>> >>>>>>>>> we
>> >>>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>>>>> wants
>> >>>>>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then
>> probably
>> >>> we
>> >>>>>>>>> should
>> >>>>>>>>>>>>>>>>> rename
>> >>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>> >>>>>>>>>>>>>>>>>> StateStoreClosedException
>> >>>>>>>>>>>>>>>>>>>>>>>>>> classes.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are
>> the
>> >>>>>>>>> possible
>> >>>>>>>>>>>>>>>>>> internal
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the
>> >>> call
>> >>>>>>>>> trace,
>> >>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>> which
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others,
>> >>> and
>> >>>>>>>> which
>> >>>>>>>>>>>>>>> ones
>> >>>>>>>>>>>>>>>>>>>> should
>> >>>>>>>>>>>>>>>>>>>>>> be
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>> >>>>>>> should
>> >>>>>>>> not
>> >>>>>>>>>>> be
>> >>>>>>>>>>>>>>>>>> wrapped
>> >>>>>>>>>>>>>>>>>>>>>> at
>> >>>>>>>>>>>>>>>>>>>>>>>>>> all
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>> >>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>> >>>>>>>>> confluence/display/KAFKA/KIP-
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>> 216%3A+IQ+should+throw+different+exceptions+for+
>> >>>>>>>>>>>>>>>>>>>> different+errors
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> --
>> >>>>>>>>>>>>>>>> -- Guozhang
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> -- Guozhang
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >
>>
>>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, Matthias,
This is second part.
> For the internal exceptions:
>
> `StateStoreClosedException` -- why can it be wrapped as
> `StreamThreadNotStartedException` ? It seems that the later would only
> be thrown by `KafkaStreams#store()` and thus would be throw directly.
Both `StateStoreClosedException` and `EmptyStateStoreException` not can be
wrapped as `StreamThreadNotStartedException`.
This is a mistaken written in the previous KIP. Thank you point this.
> A closed-exception should only happen after a store was successfully
> retrieved but cannot be queried any longer? Hence, converting/wrapping
> it into a `StateStoreMigratedException` make sense. I am also not sure,
> when a closed-exception would be wrapped by a
> `StateStoreNotAvailableException` (implying my understanding as describe
> above)?
>
> Same questions about `EmptyStateStoreException`.
>
> Thinking about both internal exceptions twice, I am wondering if it
> makes sense to have both internal exceptions at all? I have the
> impression that it make only sense to wrap them with a
> `StateStoreMigragedException`, but if they are wrapped into the same
> exception all the time, we can just remove both and throw
> `StateStoreMigratedException` directly?
After deeper thinking, I think you are right. It seems we can throw
`StateStoreMigratedException` directly.
So that we can remove `StateStoreClosedException`,
`EmptyStateStoreException` and `StateStoreNotAvailableException`.
Will update the KIP.
BTW, if we remove above three exceptions, the
`StreamThreadNotRunningException` will be the only one sub class extends
from FatalStateStoreException.
Should we remove `StreamThreadNotRunningException` and throw
`FatalStateStoreException` directly ?
> Last point: Why do we need to add?
> QueryableStoreType#setStreams(KafkaStreams streams);
> John asked this question already and you replied to it. But I am not
> sure what your answer means. Can you explain it in more detail?
The main purpose is to pass the KafkaStreams reference into
CompositeReadOnlyKeyValueStore / CompositeReadOnlySessionStore/
CompositeReadOnlyWindowStore instance.
We need check KafkaStreams state to warp InvalidStateStoreException in to
other exception(e.g., StateStoreMigratedException) when the user accesses
these read-only stores.
The original thought is to add `setStreams` method in to
QueryableStoreType. But now I think I find a better way during recent days.
This way does not need to change any public interface. So we can skip this
question. :)
I will update the KIP based on our discussion.
Thank you for help to finish the KIP!
---
Vito
On Thu, Jun 6, 2019 at 8:23 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Hi Vito,
>
> sorry for dropping this discussion on the floor a while back. I was just
> re-reading the KIP and discussion thread, and I think it is shaping out
> nicely!
>
> I like the overall hierarchy of the exception classes.
>
> Some things are still not 100% clear:
>
>
> You listed all methods that may throw an `InvalidStateStoreException`
> atm. For the new exceptions, can any exception be thrown by any method?
> It might help to understand this relationship better.
>
> For example, StreamThreadNotStartedException, seems to only make sense
> for `KafkaStreams#store()`?
>
>
> For `StreamThreadNotRunningException` should we rename it to
> `KafkaStreamsNotRunningException` ?
>
>
> The description of `StreamThreadNotRunningException` and
> `StateStoreNotAvailableException` seems to be the same? From my
> understandng, the description makes sense for
> `StreamThreadNotRunningException` -- for
> `StateStoreNotAvailableException` I was expecting/inferring from the
> name, that it would be thrown if no such store exists in the topology at
> all (ie, user passed in a invalid/wrong store name). For this case, this
> exception should be thrown only from `KafkaStreams#store()` ?
>
>
> For the internal exceptions:
>
> `StateStoreClosedException` -- why can it be wrapped as
> `StreamThreadNotStartedException` ? It seems that the later would only
> be thrown by `KafkaStreams#store()` and thus would be throw directly. A
> closed-exception should only happen after a store was successfully
> retrieved but cannot be queried any longer? Hence, converting/wrapping
> it into a `StateStoreMigratedException` make sense. I am also not sure,
> when a closed-exception would be wrapped by a
> `StateStoreNotAvailableException` (implying my understanding as describe
> above)?
>
> Same questions about `EmptyStateStoreException`.
>
> Thinking about both internal exceptions twice, I am wondering if it
> makes sense to have both internal exceptions at all? I have the
> impression that it make only sense to wrap them with a
> `StateStoreMigragedException`, but if they are wrapped into the same
> exception all the time, we can just remove both and throw
> `StateStoreMigratedException` directly?
>
>
> Last point: Why do we need to add?
>
> > QueryableStoreType#setStreams(KafkaStreams streams);
>
> John asked this question already and you replied to it. But I am not
> sure what your answer means. Can you explain it in more detail?
>
>
>
> Thanks for your patience on this KIP!
>
>
>
> -Matthias
>
>
>
>
>
>
> On 11/11/18 4:55 AM, Vito Jeng wrote:
> > Hi, Matthias,
> >
> > KIP already updated.
> >
> >> - StateStoreClosedException:
> >> will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Can you clarify the cases (ie, when will it be wrapped with the one or
> > the other)?
> >
> > For example, in the implementation(CompositeReadOnlyKeyValueStore#get),
> we
> > get all stores first, and then call ReadOnlyKeyValueStore#get to get
> value
> > in every store iteration.
> >
> > When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException
> will
> > be thrown if the state store is not open.
> > We need catch StateStoreClosedException and wrap it in different
> exception
> > type:
> > * If the stream's state is CREATED, we wrap StateStoreClosedException
> > with StreamThreadNotStartedException. User can retry until to RUNNING.
> > * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap StateStoreClosedException with
> > StateStoreMigratedException. User can rediscover the state store.
> > * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap StateStoreClosedException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> >
> >> - StateStoreIsEmptyException:
> >> I don't understand the semantic of this exception. Maybe it's a naming
> > issue?
> >
> > I think yes. :)
> > Does `EmptyStateStoreException` is better ? (already updated in the KIP)
> >
> >
> >> - StateStoreIsEmptyException:
> >> will be wrapped to StateStoreMigratedException or
> > StateStoreNotAvailableException later.
> >> Also, can you clarify the cases (ie, when will it be wrapped with the
> one
> > or the other)?
> >
> > For example, in the implementation (CompositeReadOnlyKeyValueStore#get),
> we
> > call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
> > stores. EmptyStateStoreException will be thrown when cannot find any
> store
> > and then we need catch it and wrap it in different exception type:
> > * If the stream's state is CREATED, we wrap EmptyStateStoreException
> with
> > StreamThreadNotStartedException. User can retry until to RUNNING.
> > * If the stream's state is RUNNING / REBALANCING, the state store
> should
> > be migrated, we wrap EmptyStateStoreException with
> > StateStoreMigratedException. User can rediscover the state store.
> > * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> > stream thread is not available, we wrap EmptyStateStoreException with
> > StateStoreNotAvailableException. User cannot retry when this exception is
> > thrown.
> >
> > I hope the above reply can clarify.
> >
> > The last one that was not replied was:
> >
> >> I am also wondering, if we should introduce a fatal exception
> >> `UnkownStateStoreException` to tell users that they passed in an unknown
> >> store name?
> >
> > Until now, unknown state store is not thinking about in the KIP.
> > I believe it would be very useful for users.
> >
> > Looking at the related code(WrappingStoreProvider#stores),
> > I found that I can't distinguish between the state store was migrated or
> an
> > unknown state store.
> >
> > Any thoughts?
> >
> > ---
> > Vito
> >
> >
> >
> > On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw> wrote:
> >
> >> Hi, Matthias,
> >>
> >> Sorry for the late reply.
> >>
> >>> I am wondering what the semantic impact/change is, if we introduce
> >>> `RetryableStateStoreException` and `FatalStateStoreException` that both
> >>> inherit from it. While I like the introduction of both from a high
> level
> >>> point of view, I just want to make sure it's semantically sound and
> >>> backward compatible. Atm, I think it's fine, but I want to point it out
> >>> such that everybody can think about this, too, so we can verify that
> >>> it's a natural evolving API change.
> >>
> >> Thank you for pointing this out. This's really important for public API.
> >>
> >> Just when I was replying to you, I found that KIP needs some modify.
> >> I will fix it ASAP, and then let's continue the discussion.
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
> >> wrote:
> >>
> >>> Hey Vito,
> >>>
> >>> I saw that you updated your PR, but did not reply to my last comments.
> >>> Any thoughts?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
> >>>> Glad to have you back Vito :)
> >>>>
> >>>> Some follow up thoughts:
> >>>>
> >>>> - the current `InvalidStateStoreException` is documents as being
> >>>> sometimes retry-able. From the JavaDocs:
> >>>>
> >>>>> These exceptions may be transient [...] Hence, it is valid to backoff
> >>> and retry when handling this exception.
> >>>>
> >>>> I am wondering what the semantic impact/change is, if we introduce
> >>>> `RetryableStateStoreException` and `FatalStateStoreException` that
> both
> >>>> inherit from it. While I like the introduction of both from a high
> level
> >>>> point of view, I just want to make sure it's semantically sound and
> >>>> backward compatible. Atm, I think it's fine, but I want to point it
> out
> >>>> such that everybody can think about this, too, so we can verify that
> >>>> it's a natural evolving API change.
> >>>>
> >>>> - StateStoreClosedException:
> >>>>
> >>>>> will be wrapped to StateStoreMigratedException or
> >>> StateStoreNotAvailableException later.
> >>>>
> >>>> Can you clarify the cases (ie, when will it be wrapped with the one or
> >>>> the other)?
> >>>>
> >>>> - StateStoreIsEmptyException:
> >>>>
> >>>> I don't understand the semantic of this exception. Maybe it's a naming
> >>>> issue?
> >>>>
> >>>>> will be wrapped to StateStoreMigratedException or
> >>> StateStoreNotAvailableException later.
> >>>>
> >>>> Also, can you clarify the cases (ie, when will it be wrapped with the
> >>>> one or the other)?
> >>>>
> >>>>
> >>>> I am also wondering, if we should introduce a fatal exception
> >>>> `UnkownStateStoreException` to tell users that they passed in an
> unknown
> >>>> store name?
> >>>>
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>>
> >>>> On 10/17/18 8:14 PM, vito jeng wrote:
> >>>>> Just open a PR for further discussion:
> >>>>> https://github.com/apache/kafka/pull/5814
> >>>>>
> >>>>> Any suggestion is welcome.
> >>>>> Thanks!
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>>
> >>>>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
> >>> wrote:
> >>>>>
> >>>>>> Hi John,
> >>>>>>
> >>>>>> Thanks for reviewing the KIP.
> >>>>>>
> >>>>>>> I didn't follow the addition of a new method to the
> >>> QueryableStoreType
> >>>>>>> interface. Can you elaborate why this is necessary to support the
> new
> >>>>>>> exception types?
> >>>>>>
> >>>>>> To support the new exception types, I would check stream state in
> the
> >>>>>> following classes:
> >>>>>> - CompositeReadOnlyKeyValueStore class
> >>>>>> - CompositeReadOnlySessionStore class
> >>>>>> - CompositeReadOnlyWindowStore class
> >>>>>> - DelegatingPeekingKeyValueIterator class
> >>>>>>
> >>>>>> It is also necessary to keep backward compatibility. So I plan
> passing
> >>>>>> stream
> >>>>>> instance to QueryableStoreType instance during KafkaStreams#store()
> >>>>>> invoked.
> >>>>>> It looks a most simple way, I think.
> >>>>>>
> >>>>>> It is why I add a new method to the QueryableStoreType interface. I
> >>> can
> >>>>>> understand
> >>>>>> that we should try to avoid adding the public api method. However,
> at
> >>> the
> >>>>>> moment
> >>>>>> I have no better ideas.
> >>>>>>
> >>>>>> Any thoughts?
> >>>>>>
> >>>>>>
> >>>>>>> Also, looking over your KIP again, it seems valuable to introduce
> >>>>>>> "retriable store exception" and "fatal store exception" marker
> >>> interfaces
> >>>>>>> that the various exceptions can mix in. It would be nice from a
> >>> usability
> >>>>>>> perspective to be able to just log and retry on any "retriable"
> >>> exception
> >>>>>>> and log and shutdown on any fatal exception.
> >>>>>>
> >>>>>> I agree that this is valuable to the user.
> >>>>>> I'll update the KIP.
> >>>>>>
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
> >>> wrote:
> >>>>>>
> >>>>>>> Hi Vito,
> >>>>>>>
> >>>>>>> I'm glad to hear you're well again!
> >>>>>>>
> >>>>>>> I didn't follow the addition of a new method to the
> >>> QueryableStoreType
> >>>>>>> interface. Can you elaborate why this is necessary to support the
> new
> >>>>>>> exception types?
> >>>>>>>
> >>>>>>> Also, looking over your KIP again, it seems valuable to introduce
> >>>>>>> "retriable store exception" and "fatal store exception" marker
> >>> interfaces
> >>>>>>> that the various exceptions can mix in. It would be nice from a
> >>> usability
> >>>>>>> perspective to be able to just log and retry on any "retriable"
> >>> exception
> >>>>>>> and log and shutdown on any fatal exception.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> -John
> >>>>>>>
> >>>>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com>
> >>> wrote:
> >>>>>>>
> >>>>>>>> Thanks for the explanation, that makes sense.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Guozhang
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
> >>> matthias@confluent.io
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> The scenario I had I mind was, that KS is started in one thread
> >>> while
> >>>>>>> a
> >>>>>>>>> second thread has a reference to the object to issue queries.
> >>>>>>>>>
> >>>>>>>>> If a query is issue before the "main thread" started KS, and the
> >>>>>>> "query
> >>>>>>>>> thread" knows that it will eventually get started, it can retry.
> On
> >>>>>>> the
> >>>>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
> >>>>>>> impossible
> >>>>>>>>> to issue any query against it now or in the future and thus the
> >>> error
> >>>>>>> is
> >>>>>>>>> not retryable.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
> >>>>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
> >>>>>>>>>> StreamThreadNotRunning, because I think users' handling logic
> for
> >>>>>>> the
> >>>>>>>>> third
> >>>>>>>>>> case would be likely the same as the second. Do you have some
> >>>>>>> scenarios
> >>>>>>>>>> where users may want to handle them differently?
> >>>>>>>>>>
> >>>>>>>>>> Guozhang
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
> >>>>>>>> matthias@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Sorry to hear! Get well soon!
> >>>>>>>>>>>
> >>>>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free
> to
> >>>>>>> pick
> >>>>>>>> it
> >>>>>>>>>>> up again when you find time.
> >>>>>>>>>>>
> >>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> >>> error?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
> >>> retryable
> >>>>>>>>> error.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is
> not
> >>>>>>>>>>> RUNNING. The
> >>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>>>>
> >>>>>>>>>>> I see. If this is the intention, than I would suggest to have
> two
> >>>>>>> (or
> >>>>>>>>>>> maybe three) different exceptions:
> >>>>>>>>>>>
> >>>>>>>>>>> - StreamThreadRebalancingException (retryable)
> >>>>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
> >>>>>>>>>>> PENDING_SHUTDOWN or DEAD
> >>>>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
> >>>>>>>>>>>
> >>>>>>>>>>> The last one is tricky and could also be merged into one of the
> >>>>>>> first
> >>>>>>>>>>> two, depending if you want to argue that it's retryable or not.
> >>>>>>> (Just
> >>>>>>>>>>> food for though -- not sure what others think.)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -Matthias
> >>>>>>>>>>>
> >>>>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
> >>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thank you for your assistance.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Unfortunately, there is no further progress.
> >>>>>>>>>>>> About seven weeks ago, I was injured in sports. I had a broken
> >>>>>>> wrist
> >>>>>>>> on
> >>>>>>>>>>>> my left wrist.
> >>>>>>>>>>>> Many jobs are affected, including this KIP and implementation.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
> Why
> >>>>>>> do
> >>>>>>>> we
> >>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
> >>>>>>> really
> >>>>>>>>> need
> >>>>>>>>>>>>> them? Can't we just throw the correct exception directly
> >>> instead
> >>>>>>> of
> >>>>>>>>>>>>> wrapping it later?
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think you may be right. As I say in the previous:
> >>>>>>>>>>>> "The original idea is that we can distinguish different state
> >>>>>>> store
> >>>>>>>>>>>> exception for different handling. But to be honest, I am not
> >>> quite
> >>>>>>>> sure
> >>>>>>>>>>>> this is necessary. Maybe have some change during
> >>> implementation."
> >>>>>>>>>>>>
> >>>>>>>>>>>> During the implementation, I also feel we maybe not need
> wrapper
> >>>>>>> it.
> >>>>>>>>>>>> We can just throw the correctly directly.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> error?
> >>>>>>>>>>>>
> >>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
> retryable
> >>>>>>>> error.
> >>>>>>>>>>>>
> >>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
> >>>>>>>>> RUNNING.
> >>>>>>>>>>> The
> >>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
> >>> semantics
> >>>>>>> is
> >>>>>>>>>>>> unclear to me atm.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
> >>>>>>>> retryable
> >>>>>>>>>>>> error?
> >>>>>>>>>>>>
> >>>>>>>>>>>> These two comments will be answered in another mail.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
> >>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments.
> Why
> >>>>>>> do
> >>>>>>>> we
> >>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
> >>>>>>> really
> >>>>>>>>> need
> >>>>>>>>>>>>> them? Can't we just throw the correct exception directly
> >>> instead
> >>>>>>> of
> >>>>>>>>>>>>> wrapping it later?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
> >>> semantics
> >>>>>>> is
> >>>>>>>>>>>>> unclear to me atm.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
> error?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
> >>>>>>>> retryable
> >>>>>>>>>>>>> error?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
> >>> key,
> >>>>>>>> long
> >>>>>>>>>>>>> time); that should be added
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Overall I like the KIP but some details are still unclear.
> >>> Maybe
> >>>>>>> it
> >>>>>>>>>>>>> might help if you open an PR in parallel?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
> >>>>>>>>>>>>>> Hi, Guozhang,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi, Bill,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'll try to make some update to make the KIP better.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
> >>> bbejeck@gmail.com
> >>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> At this point, the only thing I would change is possibly
> >>>>>>> removing
> >>>>>>>>> the
> >>>>>>>>>>>>>>> listing of all methods called by the user and the listing
> of
> >>>>>>> all
> >>>>>>>>> store
> >>>>>>>>>>>>>>> types and focus on what states result in which exceptions
> >>>>>>> thrown
> >>>>>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>> Bill
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
> >>>>>>>> wangguoz@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me. I'm
> +1
> >>>>>>> on
> >>>>>>>> the
> >>>>>>>>>>>>> KIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> About the base class InvalidStateStoreException itself,
> I'd
> >>>>>>>>> actually
> >>>>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as part
> >>> of
> >>>>>>> the
> >>>>>>>>>>>>> public
> >>>>>>>>>>>>>>>> API, for people who do not want to handle these cases
> >>>>>>> differently
> >>>>>>>>> (if
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all
> three
> >>>>>>>>>>> exceptions
> >>>>>>>>>>>>>>>> one-by-one).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
> >>>>>>> john@confluent.io
> >>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think it's much nicer to give callers different
> >>> exceptions
> >>>>>>> to
> >>>>>>>>> tell
> >>>>>>>>>>>>>>> them
> >>>>>>>>>>>>>>>>> whether the state store got migrated, whether it's still
> >>>>>>>>>>> initializing,
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> whether there's some unrecoverable error.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
> >>>>>>>>> non-user-facing
> >>>>>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>> such as what exceptions we will throw internally. The KIP
> >>> is
> >>>>>>>>>>> primarily
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> discuss public interface changes.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> You might consider simply removing all the internal
> details
> >>>>>>> from
> >>>>>>>>> the
> >>>>>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
> >>>>>>> smaller
> >>>>>>>>> and
> >>>>>>>>>>>>>>>> easier
> >>>>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
> >>>>>>> internal
> >>>>>>>>>>>>> details
> >>>>>>>>>>>>>>>>> when it comes to implementation.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I like your decision to have your refined exceptions
> extend
> >>>>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
> >>> compatibility.
> >>>>>>>> Since
> >>>>>>>>>>> we
> >>>>>>>>>>>>>>>> want
> >>>>>>>>>>>>>>>>> to encourage callers to catch the more specific
> exceptions,
> >>>>>>> and
> >>>>>>>> we
> >>>>>>>>>>>>>>> don't
> >>>>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
> >>>>>>> anymore,
> >>>>>>>> you
> >>>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
> >>>>>>>>>>>>>>> InvalidStateStoreException.
> >>>>>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
> >>>>>>>> exception
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>> open
> >>>>>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
> >>>>>>> entirely
> >>>>>>>>> in a
> >>>>>>>>>>>>>>>> future
> >>>>>>>>>>>>>>>>> release.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions
> explicit?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for the feedback!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
> >>> not.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Got it!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
> used?
> >>> Do
> >>>>>>>> you
> >>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> >>>>>>>> anymore,
> >>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
> >>> understanding).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
> >>>>>>> thrown
> >>>>>>>>> at
> >>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>> anymore,
> >>>>>>>>>>>>>>>>>>> but only new sub-classes.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
> >>>>>>> exception
> >>>>>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
> >>> will
> >>>>>>>>> throw
> >>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
> >>>>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
> >>>>>>>>>>>>>>>>>>> `StateStoreRetryableException` or
> >>> `StateStoreFailException`
> >>>>>>>>> during
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
> >>> exceptions".
> >>>>>>>> It's
> >>>>>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
> >>>>>>>>> distinguish
> >>>>>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
> >>>>>>>>>>>>>>>>>>> The original idea is that we can distinguish different
> >>>>>>> state
> >>>>>>>>> store
> >>>>>>>>>>>>>>>>>>> exception for
> >>>>>>>>>>>>>>>>>>> different handling.
> >>>>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is
> necessary.
> >>>>>>> Maybe
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>>>>>> some change during implementation.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Does it make sense?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Thanks for the update Vito!
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
> >>> not.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be
> used?
> >>> Do
> >>>>>>>> you
> >>>>>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> >>>>>>>> anymore,
> >>>>>>>>>>>>>>> but
> >>>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
> >>> understanding).
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
> >>>>>>> exception
> >>>>>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
> >>> exceptions".
> >>>>>>>> It's
> >>>>>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks for the review.
> >>>>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
> >>>>>>> pause...
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
> >>> code.
> >>>>>>>> It's
> >>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are affected.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
> >>>>>>> `Changes
> >>>>>>>> in
> >>>>>>>>>>>>>>> call
> >>>>>>>>>>>>>>>>>>>> trace`.
> >>>>>>>>>>>>>>>>>>>>> Do you think it should be removed?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP,
> the
> >>>>>>> main
> >>>>>>>>>>>>>>> change
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
> >>>>>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
> >>> things.
> >>>>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions
> extend
> >>>>>>> from
> >>>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Some questions:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
> >>>>>>> `InvalidStateStoreException`
> >>>>>>>> be
> >>>>>>>>>>>>>>> used
> >>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>> this purpose?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Does this question miss some word?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - What the superclass of
> >>>>>>> `StateStoreStreamThreadNotRunni
> >>>>>>>>>>>>>>>>> ngException`
> >>>>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
> >>> exception
> >>>>>>> ?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
> >>> modify
> >>>>>>>> some
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
> >>>>>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the
> state
> >>>>>>> both
> >>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams
> thread
> >>>>>>> and
> >>>>>>>>> state
> >>>>>>>>>>>>>>>>> store,
> >>>>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax
> <
> >>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is
> >>> not
> >>>>>>>>> hosted
> >>>>>>>>>>>>>>> an
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
> >>>>>>> state
> >>>>>>>>>>>>>>>> rebalance
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
> >>>>>>> store.
> >>>>>>>>> For
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> second
> >>>>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
> >>> finished.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
> >>>>>>> PENDING_SHUTDOWN, or
> >>>>>>>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
> >>>>>>>> rediscover
> >>>>>>>>> or
> >>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
> >>>>>>> user
> >>>>>>>> can
> >>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>>> plain
> >>>>>>>>>>>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN
> >>> or
> >>>>>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the
> >>> user
> >>>>>>>> needs
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>> rediscover
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J.
> Sax
> >>> <
> >>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
> >>>>>>> idea of
> >>>>>>>>> the
> >>>>>>>>>>>>>>>> Jira
> >>>>>>>>>>>>>>>>>>>> was,
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
> >>> "recovery"
> >>>>>>>>>>>>>>>> strategies
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently
> recreated,
> >>> a
> >>>>>>>> user
> >>>>>>>>>>>>>>> just
> >>>>>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>> wait
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other
> >>> hand,
> >>>>>>> if
> >>>>>>>> a
> >>>>>>>>>>>>>>> store
> >>>>>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>>>>>>> migrated
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to
> rediscover
> >>>>>>> the
> >>>>>>>>> store
> >>>>>>>>>>>>>>>>>> instead
> >>>>>>>>>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
> >>>>>>> talks
> >>>>>>>>> about
> >>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>> ideas
> >>>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
> >>> figure
> >>>>>>>> out
> >>>>>>>>>>>>>>> first
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
> >>>>>>> users, if
> >>>>>>>>> we
> >>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>>>> wants
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then
> probably
> >>> we
> >>>>>>>>> should
> >>>>>>>>>>>>>>>>> rename
> >>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>>>>>>>>>>>>>>> StateStoreClosedException
> >>>>>>>>>>>>>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are
> the
> >>>>>>>>> possible
> >>>>>>>>>>>>>>>>>> internal
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the
> >>> call
> >>>>>>>>> trace,
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others,
> >>> and
> >>>>>>>> which
> >>>>>>>>>>>>>>> ones
> >>>>>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
> >>>>>>> should
> >>>>>>>> not
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> wrapped
> >>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> >>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
> >>>>>>>>> confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>>>>>>>>>>>>>>> different+errors
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> -- Guozhang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> -- Guozhang
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hi Vito,
sorry for dropping this discussion on the floor a while back. I was just
re-reading the KIP and discussion thread, and I think it is shaping out
nicely!
I like the overall hierarchy of the exception classes.
Some things are still not 100% clear:
You listed all methods that may throw an `InvalidStateStoreException`
atm. For the new exceptions, can any exception be thrown by any method?
It might help to understand this relationship better.
For example, StreamThreadNotStartedException, seems to only make sense
for `KafkaStreams#store()`?
For `StreamThreadNotRunningException` should we rename it to
`KafkaStreamsNotRunningException` ?
The description of `StreamThreadNotRunningException` and
`StateStoreNotAvailableException` seems to be the same? From my
understandng, the description makes sense for
`StreamThreadNotRunningException` -- for
`StateStoreNotAvailableException` I was expecting/inferring from the
name, that it would be thrown if no such store exists in the topology at
all (ie, user passed in a invalid/wrong store name). For this case, this
exception should be thrown only from `KafkaStreams#store()` ?
For the internal exceptions:
`StateStoreClosedException` -- why can it be wrapped as
`StreamThreadNotStartedException` ? It seems that the later would only
be thrown by `KafkaStreams#store()` and thus would be throw directly. A
closed-exception should only happen after a store was successfully
retrieved but cannot be queried any longer? Hence, converting/wrapping
it into a `StateStoreMigratedException` make sense. I am also not sure,
when a closed-exception would be wrapped by a
`StateStoreNotAvailableException` (implying my understanding as describe
above)?
Same questions about `EmptyStateStoreException`.
Thinking about both internal exceptions twice, I am wondering if it
makes sense to have both internal exceptions at all? I have the
impression that it make only sense to wrap them with a
`StateStoreMigragedException`, but if they are wrapped into the same
exception all the time, we can just remove both and throw
`StateStoreMigratedException` directly?
Last point: Why do we need to add?
> QueryableStoreType#setStreams(KafkaStreams streams);
John asked this question already and you replied to it. But I am not
sure what your answer means. Can you explain it in more detail?
Thanks for your patience on this KIP!
-Matthias
On 11/11/18 4:55 AM, Vito Jeng wrote:
> Hi, Matthias,
>
> KIP already updated.
>
>> - StateStoreClosedException:
>> will be wrapped to StateStoreMigratedException or
> StateStoreNotAvailableException later.
>> Can you clarify the cases (ie, when will it be wrapped with the one or
> the other)?
>
> For example, in the implementation(CompositeReadOnlyKeyValueStore#get), we
> get all stores first, and then call ReadOnlyKeyValueStore#get to get value
> in every store iteration.
>
> When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException will
> be thrown if the state store is not open.
> We need catch StateStoreClosedException and wrap it in different exception
> type:
> * If the stream's state is CREATED, we wrap StateStoreClosedException
> with StreamThreadNotStartedException. User can retry until to RUNNING.
> * If the stream's state is RUNNING / REBALANCING, the state store should
> be migrated, we wrap StateStoreClosedException with
> StateStoreMigratedException. User can rediscover the state store.
> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> stream thread is not available, we wrap StateStoreClosedException with
> StateStoreNotAvailableException. User cannot retry when this exception is
> thrown.
>
>
>> - StateStoreIsEmptyException:
>> I don't understand the semantic of this exception. Maybe it's a naming
> issue?
>
> I think yes. :)
> Does `EmptyStateStoreException` is better ? (already updated in the KIP)
>
>
>> - StateStoreIsEmptyException:
>> will be wrapped to StateStoreMigratedException or
> StateStoreNotAvailableException later.
>> Also, can you clarify the cases (ie, when will it be wrapped with the one
> or the other)?
>
> For example, in the implementation (CompositeReadOnlyKeyValueStore#get), we
> call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
> stores. EmptyStateStoreException will be thrown when cannot find any store
> and then we need catch it and wrap it in different exception type:
> * If the stream's state is CREATED, we wrap EmptyStateStoreException with
> StreamThreadNotStartedException. User can retry until to RUNNING.
> * If the stream's state is RUNNING / REBALANCING, the state store should
> be migrated, we wrap EmptyStateStoreException with
> StateStoreMigratedException. User can rediscover the state store.
> * If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
> stream thread is not available, we wrap EmptyStateStoreException with
> StateStoreNotAvailableException. User cannot retry when this exception is
> thrown.
>
> I hope the above reply can clarify.
>
> The last one that was not replied was:
>
>> I am also wondering, if we should introduce a fatal exception
>> `UnkownStateStoreException` to tell users that they passed in an unknown
>> store name?
>
> Until now, unknown state store is not thinking about in the KIP.
> I believe it would be very useful for users.
>
> Looking at the related code(WrappingStoreProvider#stores),
> I found that I can't distinguish between the state store was migrated or an
> unknown state store.
>
> Any thoughts?
>
> ---
> Vito
>
>
>
> On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw> wrote:
>
>> Hi, Matthias,
>>
>> Sorry for the late reply.
>>
>>> I am wondering what the semantic impact/change is, if we introduce
>>> `RetryableStateStoreException` and `FatalStateStoreException` that both
>>> inherit from it. While I like the introduction of both from a high level
>>> point of view, I just want to make sure it's semantically sound and
>>> backward compatible. Atm, I think it's fine, but I want to point it out
>>> such that everybody can think about this, too, so we can verify that
>>> it's a natural evolving API change.
>>
>> Thank you for pointing this out. This's really important for public API.
>>
>> Just when I was replying to you, I found that KIP needs some modify.
>> I will fix it ASAP, and then let's continue the discussion.
>>
>> ---
>> Vito
>>
>>
>> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
>> wrote:
>>
>>> Hey Vito,
>>>
>>> I saw that you updated your PR, but did not reply to my last comments.
>>> Any thoughts?
>>>
>>>
>>> -Matthias
>>>
>>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
>>>> Glad to have you back Vito :)
>>>>
>>>> Some follow up thoughts:
>>>>
>>>> - the current `InvalidStateStoreException` is documents as being
>>>> sometimes retry-able. From the JavaDocs:
>>>>
>>>>> These exceptions may be transient [...] Hence, it is valid to backoff
>>> and retry when handling this exception.
>>>>
>>>> I am wondering what the semantic impact/change is, if we introduce
>>>> `RetryableStateStoreException` and `FatalStateStoreException` that both
>>>> inherit from it. While I like the introduction of both from a high level
>>>> point of view, I just want to make sure it's semantically sound and
>>>> backward compatible. Atm, I think it's fine, but I want to point it out
>>>> such that everybody can think about this, too, so we can verify that
>>>> it's a natural evolving API change.
>>>>
>>>> - StateStoreClosedException:
>>>>
>>>>> will be wrapped to StateStoreMigratedException or
>>> StateStoreNotAvailableException later.
>>>>
>>>> Can you clarify the cases (ie, when will it be wrapped with the one or
>>>> the other)?
>>>>
>>>> - StateStoreIsEmptyException:
>>>>
>>>> I don't understand the semantic of this exception. Maybe it's a naming
>>>> issue?
>>>>
>>>>> will be wrapped to StateStoreMigratedException or
>>> StateStoreNotAvailableException later.
>>>>
>>>> Also, can you clarify the cases (ie, when will it be wrapped with the
>>>> one or the other)?
>>>>
>>>>
>>>> I am also wondering, if we should introduce a fatal exception
>>>> `UnkownStateStoreException` to tell users that they passed in an unknown
>>>> store name?
>>>>
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>>
>>>> On 10/17/18 8:14 PM, vito jeng wrote:
>>>>> Just open a PR for further discussion:
>>>>> https://github.com/apache/kafka/pull/5814
>>>>>
>>>>> Any suggestion is welcome.
>>>>> Thanks!
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>>
>>>>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
>>> wrote:
>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> Thanks for reviewing the KIP.
>>>>>>
>>>>>>> I didn't follow the addition of a new method to the
>>> QueryableStoreType
>>>>>>> interface. Can you elaborate why this is necessary to support the new
>>>>>>> exception types?
>>>>>>
>>>>>> To support the new exception types, I would check stream state in the
>>>>>> following classes:
>>>>>> - CompositeReadOnlyKeyValueStore class
>>>>>> - CompositeReadOnlySessionStore class
>>>>>> - CompositeReadOnlyWindowStore class
>>>>>> - DelegatingPeekingKeyValueIterator class
>>>>>>
>>>>>> It is also necessary to keep backward compatibility. So I plan passing
>>>>>> stream
>>>>>> instance to QueryableStoreType instance during KafkaStreams#store()
>>>>>> invoked.
>>>>>> It looks a most simple way, I think.
>>>>>>
>>>>>> It is why I add a new method to the QueryableStoreType interface. I
>>> can
>>>>>> understand
>>>>>> that we should try to avoid adding the public api method. However, at
>>> the
>>>>>> moment
>>>>>> I have no better ideas.
>>>>>>
>>>>>> Any thoughts?
>>>>>>
>>>>>>
>>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>>>>> "retriable store exception" and "fatal store exception" marker
>>> interfaces
>>>>>>> that the various exceptions can mix in. It would be nice from a
>>> usability
>>>>>>> perspective to be able to just log and retry on any "retriable"
>>> exception
>>>>>>> and log and shutdown on any fatal exception.
>>>>>>
>>>>>> I agree that this is valuable to the user.
>>>>>> I'll update the KIP.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Vito
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
>>> wrote:
>>>>>>
>>>>>>> Hi Vito,
>>>>>>>
>>>>>>> I'm glad to hear you're well again!
>>>>>>>
>>>>>>> I didn't follow the addition of a new method to the
>>> QueryableStoreType
>>>>>>> interface. Can you elaborate why this is necessary to support the new
>>>>>>> exception types?
>>>>>>>
>>>>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>>>>> "retriable store exception" and "fatal store exception" marker
>>> interfaces
>>>>>>> that the various exceptions can mix in. It would be nice from a
>>> usability
>>>>>>> perspective to be able to just log and retry on any "retriable"
>>> exception
>>>>>>> and log and shutdown on any fatal exception.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -John
>>>>>>>
>>>>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com>
>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the explanation, that makes sense.
>>>>>>>>
>>>>>>>>
>>>>>>>> Guozhang
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
>>> matthias@confluent.io
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The scenario I had I mind was, that KS is started in one thread
>>> while
>>>>>>> a
>>>>>>>>> second thread has a reference to the object to issue queries.
>>>>>>>>>
>>>>>>>>> If a query is issue before the "main thread" started KS, and the
>>>>>>> "query
>>>>>>>>> thread" knows that it will eventually get started, it can retry. On
>>>>>>> the
>>>>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>>>>>>> impossible
>>>>>>>>> to issue any query against it now or in the future and thus the
>>> error
>>>>>>> is
>>>>>>>>> not retryable.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
>>>>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
>>>>>>>>>> StreamThreadNotRunning, because I think users' handling logic for
>>>>>>> the
>>>>>>>>> third
>>>>>>>>>> case would be likely the same as the second. Do you have some
>>>>>>> scenarios
>>>>>>>>>> where users may want to handle them differently?
>>>>>>>>>>
>>>>>>>>>> Guozhang
>>>>>>>>>>
>>>>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>>>>>>>> matthias@confluent.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sorry to hear! Get well soon!
>>>>>>>>>>>
>>>>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free to
>>>>>>> pick
>>>>>>>> it
>>>>>>>>>>> up again when you find time.
>>>>>>>>>>>
>>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>>> error?
>>>>>>>>>>>>>
>>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>>> retryable
>>>>>>>>> error.
>>>>>>>>>>>>>
>>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>>>>>>>>>>> RUNNING. The
>>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>>>>
>>>>>>>>>>> I see. If this is the intention, than I would suggest to have two
>>>>>>> (or
>>>>>>>>>>> maybe three) different exceptions:
>>>>>>>>>>>
>>>>>>>>>>> - StreamThreadRebalancingException (retryable)
>>>>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
>>>>>>>>>>> PENDING_SHUTDOWN or DEAD
>>>>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
>>>>>>>>>>>
>>>>>>>>>>> The last one is tricky and could also be merged into one of the
>>>>>>> first
>>>>>>>>>>> two, depending if you want to argue that it's retryable or not.
>>>>>>> (Just
>>>>>>>>>>> food for though -- not sure what others think.)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for your assistance.
>>>>>>>>>>>>
>>>>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>>>>
>>>>>>>>>>>> Unfortunately, there is no further progress.
>>>>>>>>>>>> About seven weeks ago, I was injured in sports. I had a broken
>>>>>>> wrist
>>>>>>>> on
>>>>>>>>>>>> my left wrist.
>>>>>>>>>>>> Many jobs are affected, including this KIP and implementation.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>>>>>>> do
>>>>>>>> we
>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>>>>> really
>>>>>>>>> need
>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>>> instead
>>>>>>> of
>>>>>>>>>>>>> wrapping it later?
>>>>>>>>>>>>
>>>>>>>>>>>> I think you may be right. As I say in the previous:
>>>>>>>>>>>> "The original idea is that we can distinguish different state
>>>>>>> store
>>>>>>>>>>>> exception for different handling. But to be honest, I am not
>>> quite
>>>>>>>> sure
>>>>>>>>>>>> this is necessary. Maybe have some change during
>>> implementation."
>>>>>>>>>>>>
>>>>>>>>>>>> During the implementation, I also feel we maybe not need wrapper
>>>>>>> it.
>>>>>>>>>>>> We can just throw the correctly directly.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>>>>>
>>>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
>>>>>>>> error.
>>>>>>>>>>>>
>>>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>>>>>>>>> RUNNING.
>>>>>>>>>>> The
>>>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>>> semantics
>>>>>>> is
>>>>>>>>>>>> unclear to me atm.
>>>>>>>>>>>>
>>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>>>>> retryable
>>>>>>>>>>>> error?
>>>>>>>>>>>>
>>>>>>>>>>>> These two comments will be answered in another mail.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Vito
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>
>>>>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>>>>>>> do
>>>>>>>> we
>>>>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>>>>> really
>>>>>>>>> need
>>>>>>>>>>>>> them? Can't we just throw the correct exception directly
>>> instead
>>>>>>> of
>>>>>>>>>>>>> wrapping it later?
>>>>>>>>>>>>>
>>>>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>>> semantics
>>>>>>> is
>>>>>>>>>>>>> unclear to me atm.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>>>>>>
>>>>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>>>>> retryable
>>>>>>>>>>>>> error?
>>>>>>>>>>>>>
>>>>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
>>> key,
>>>>>>>> long
>>>>>>>>>>>>> time); that should be added
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Overall I like the KIP but some details are still unclear.
>>> Maybe
>>>>>>> it
>>>>>>>>>>>>> might help if you open an PR in parallel?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>>>>>>>>>>>>>> Hi, Guozhang,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi, Bill,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'll try to make some update to make the KIP better.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
>>> bbejeck@gmail.com
>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At this point, the only thing I would change is possibly
>>>>>>> removing
>>>>>>>>> the
>>>>>>>>>>>>>>> listing of all methods called by the user and the listing of
>>>>>>> all
>>>>>>>>> store
>>>>>>>>>>>>>>> types and focus on what states result in which exceptions
>>>>>>> thrown
>>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Bill
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>>>>>>>> wangguoz@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me. I'm +1
>>>>>>> on
>>>>>>>> the
>>>>>>>>>>>>> KIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About the base class InvalidStateStoreException itself, I'd
>>>>>>>>> actually
>>>>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as part
>>> of
>>>>>>> the
>>>>>>>>>>>>> public
>>>>>>>>>>>>>>>> API, for people who do not want to handle these cases
>>>>>>> differently
>>>>>>>>> (if
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all three
>>>>>>>>>>> exceptions
>>>>>>>>>>>>>>>> one-by-one).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>>>>>>> john@confluent.io
>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think it's much nicer to give callers different
>>> exceptions
>>>>>>> to
>>>>>>>>> tell
>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>> whether the state store got migrated, whether it's still
>>>>>>>>>>> initializing,
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> whether there's some unrecoverable error.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
>>>>>>>>> non-user-facing
>>>>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>> such as what exceptions we will throw internally. The KIP
>>> is
>>>>>>>>>>> primarily
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> discuss public interface changes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You might consider simply removing all the internal details
>>>>>>> from
>>>>>>>>> the
>>>>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
>>>>>>> smaller
>>>>>>>>> and
>>>>>>>>>>>>>>>> easier
>>>>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
>>>>>>> internal
>>>>>>>>>>>>> details
>>>>>>>>>>>>>>>>> when it comes to implementation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I like your decision to have your refined exceptions extend
>>>>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
>>> compatibility.
>>>>>>>> Since
>>>>>>>>>>> we
>>>>>>>>>>>>>>>> want
>>>>>>>>>>>>>>>>> to encourage callers to catch the more specific exceptions,
>>>>>>> and
>>>>>>>> we
>>>>>>>>>>>>>>> don't
>>>>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
>>>>>>> anymore,
>>>>>>>> you
>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
>>>>>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
>>>>>>>> exception
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
>>>>>>> entirely
>>>>>>>>> in a
>>>>>>>>>>>>>>>> future
>>>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions explicit?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>>> not.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Got it!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used?
>>> Do
>>>>>>>> you
>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>>>>>>>> anymore,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>>> understanding).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
>>>>>>> thrown
>>>>>>>>> at
>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>>>>> but only new sub-classes.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>>>>> exception
>>>>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
>>> will
>>>>>>>>> throw
>>>>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
>>>>>>>>>>>>>>>>>>> `StateStoreRetryableException` or
>>> `StateStoreFailException`
>>>>>>>>> during
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>>> exceptions".
>>>>>>>> It's
>>>>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>>>>>>>>> distinguish
>>>>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
>>>>>>>>>>>>>>>>>>> The original idea is that we can distinguish different
>>>>>>> state
>>>>>>>>> store
>>>>>>>>>>>>>>>>>>> exception for
>>>>>>>>>>>>>>>>>>> different handling.
>>>>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is necessary.
>>>>>>> Maybe
>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>> some change during implementation.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Does it make sense?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the update Vito!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>>> not.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used?
>>> Do
>>>>>>>> you
>>>>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>>>>>>>> anymore,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>>> understanding).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>>>>> exception
>>>>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>>> exceptions".
>>>>>>>> It's
>>>>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for the review.
>>>>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>>>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>>>>>>> pause...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
>>> code.
>>>>>>>> It's
>>>>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are affected.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
>>>>>>> `Changes
>>>>>>>> in
>>>>>>>>>>>>>>> call
>>>>>>>>>>>>>>>>>>>> trace`.
>>>>>>>>>>>>>>>>>>>>> Do you think it should be removed?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP, the
>>>>>>> main
>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
>>> things.
>>>>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions extend
>>>>>>> from
>>>>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Some questions:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
>>>>>>> `InvalidStateStoreException`
>>>>>>>> be
>>>>>>>>>>>>>>> used
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> this purpose?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Does this question miss some word?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - What the superclass of
>>>>>>> `StateStoreStreamThreadNotRunni
>>>>>>>>>>>>>>>>> ngException`
>>>>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
>>> exception
>>>>>>> ?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
>>> modify
>>>>>>>> some
>>>>>>>>>>>>>>> code
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
>>>>>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the state
>>>>>>> both
>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread
>>>>>>> and
>>>>>>>>> state
>>>>>>>>>>>>>>>>> store,
>>>>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is
>>> not
>>>>>>>>> hosted
>>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>>>>>>> state
>>>>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
>>>>>>> store.
>>>>>>>>> For
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
>>> finished.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>>>>>>> PENDING_SHUTDOWN, or
>>>>>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>>>>>>>> rediscover
>>>>>>>>> or
>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
>>>>>>> user
>>>>>>>> can
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>>> plain
>>>>>>>>>>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN
>>> or
>>>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the
>>> user
>>>>>>>> needs
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> rediscover
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax
>>> <
>>>>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>>>>>>> idea of
>>>>>>>>> the
>>>>>>>>>>>>>>>> Jira
>>>>>>>>>>>>>>>>>>>> was,
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
>>> "recovery"
>>>>>>>>>>>>>>>> strategies
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated,
>>> a
>>>>>>>> user
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other
>>> hand,
>>>>>>> if
>>>>>>>> a
>>>>>>>>>>>>>>> store
>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover
>>>>>>> the
>>>>>>>>> store
>>>>>>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>>>>>>> talks
>>>>>>>>> about
>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>> ideas
>>>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
>>> figure
>>>>>>>> out
>>>>>>>>>>>>>>> first
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>>>>>>> users, if
>>>>>>>>> we
>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably
>>> we
>>>>>>>>> should
>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>>>>>>>>>>>>>>> StateStoreClosedException
>>>>>>>>>>>>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
>>>>>>>>> possible
>>>>>>>>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the
>>> call
>>>>>>>>> trace,
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others,
>>> and
>>>>>>>> which
>>>>>>>>>>>>>>> ones
>>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>>>>>>> should
>>>>>>>> not
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> wrapped
>>>>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>>>>>>>>> confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>>>>>>>>>>>>>>> different+errors
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -- Guozhang
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, Matthias,
KIP already updated.
> - StateStoreClosedException:
> will be wrapped to StateStoreMigratedException or
StateStoreNotAvailableException later.
> Can you clarify the cases (ie, when will it be wrapped with the one or
the other)?
For example, in the implementation(CompositeReadOnlyKeyValueStore#get), we
get all stores first, and then call ReadOnlyKeyValueStore#get to get value
in every store iteration.
When calling ReadOnlyKeyValueStore#get, the StateStoreClosedException will
be thrown if the state store is not open.
We need catch StateStoreClosedException and wrap it in different exception
type:
* If the stream's state is CREATED, we wrap StateStoreClosedException
with StreamThreadNotStartedException. User can retry until to RUNNING.
* If the stream's state is RUNNING / REBALANCING, the state store should
be migrated, we wrap StateStoreClosedException with
StateStoreMigratedException. User can rediscover the state store.
* If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
stream thread is not available, we wrap StateStoreClosedException with
StateStoreNotAvailableException. User cannot retry when this exception is
thrown.
> - StateStoreIsEmptyException:
> I don't understand the semantic of this exception. Maybe it's a naming
issue?
I think yes. :)
Does `EmptyStateStoreException` is better ? (already updated in the KIP)
> - StateStoreIsEmptyException:
> will be wrapped to StateStoreMigratedException or
StateStoreNotAvailableException later.
> Also, can you clarify the cases (ie, when will it be wrapped with the one
or the other)?
For example, in the implementation (CompositeReadOnlyKeyValueStore#get), we
call StateStoreProvider#stores (WrappingStoreProvider#stores) to get all
stores. EmptyStateStoreException will be thrown when cannot find any store
and then we need catch it and wrap it in different exception type:
* If the stream's state is CREATED, we wrap EmptyStateStoreException with
StreamThreadNotStartedException. User can retry until to RUNNING.
* If the stream's state is RUNNING / REBALANCING, the state store should
be migrated, we wrap EmptyStateStoreException with
StateStoreMigratedException. User can rediscover the state store.
* If the stream's state is PENDING_SHUTDOWN / NOT_RUNNING / ERROR, the
stream thread is not available, we wrap EmptyStateStoreException with
StateStoreNotAvailableException. User cannot retry when this exception is
thrown.
I hope the above reply can clarify.
The last one that was not replied was:
> I am also wondering, if we should introduce a fatal exception
> `UnkownStateStoreException` to tell users that they passed in an unknown
> store name?
Until now, unknown state store is not thinking about in the KIP.
I believe it would be very useful for users.
Looking at the related code(WrappingStoreProvider#stores),
I found that I can't distinguish between the state store was migrated or an
unknown state store.
Any thoughts?
---
Vito
On Sun, Nov 11, 2018 at 5:31 PM Vito Jeng <vi...@is-land.com.tw> wrote:
> Hi, Matthias,
>
> Sorry for the late reply.
>
> > I am wondering what the semantic impact/change is, if we introduce
> > `RetryableStateStoreException` and `FatalStateStoreException` that both
> > inherit from it. While I like the introduction of both from a high level
> > point of view, I just want to make sure it's semantically sound and
> > backward compatible. Atm, I think it's fine, but I want to point it out
> > such that everybody can think about this, too, so we can verify that
> > it's a natural evolving API change.
>
> Thank you for pointing this out. This's really important for public API.
>
> Just when I was replying to you, I found that KIP needs some modify.
> I will fix it ASAP, and then let's continue the discussion.
>
> ---
> Vito
>
>
> On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Hey Vito,
>>
>> I saw that you updated your PR, but did not reply to my last comments.
>> Any thoughts?
>>
>>
>> -Matthias
>>
>> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
>> > Glad to have you back Vito :)
>> >
>> > Some follow up thoughts:
>> >
>> > - the current `InvalidStateStoreException` is documents as being
>> > sometimes retry-able. From the JavaDocs:
>> >
>> >> These exceptions may be transient [...] Hence, it is valid to backoff
>> and retry when handling this exception.
>> >
>> > I am wondering what the semantic impact/change is, if we introduce
>> > `RetryableStateStoreException` and `FatalStateStoreException` that both
>> > inherit from it. While I like the introduction of both from a high level
>> > point of view, I just want to make sure it's semantically sound and
>> > backward compatible. Atm, I think it's fine, but I want to point it out
>> > such that everybody can think about this, too, so we can verify that
>> > it's a natural evolving API change.
>> >
>> > - StateStoreClosedException:
>> >
>> >> will be wrapped to StateStoreMigratedException or
>> StateStoreNotAvailableException later.
>> >
>> > Can you clarify the cases (ie, when will it be wrapped with the one or
>> > the other)?
>> >
>> > - StateStoreIsEmptyException:
>> >
>> > I don't understand the semantic of this exception. Maybe it's a naming
>> > issue?
>> >
>> >> will be wrapped to StateStoreMigratedException or
>> StateStoreNotAvailableException later.
>> >
>> > Also, can you clarify the cases (ie, when will it be wrapped with the
>> > one or the other)?
>> >
>> >
>> > I am also wondering, if we should introduce a fatal exception
>> > `UnkownStateStoreException` to tell users that they passed in an unknown
>> > store name?
>> >
>> >
>> >
>> > -Matthias
>> >
>> >
>> >
>> > On 10/17/18 8:14 PM, vito jeng wrote:
>> >> Just open a PR for further discussion:
>> >> https://github.com/apache/kafka/pull/5814
>> >>
>> >> Any suggestion is welcome.
>> >> Thanks!
>> >>
>> >> ---
>> >> Vito
>> >>
>> >>
>> >> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw>
>> wrote:
>> >>
>> >>> Hi John,
>> >>>
>> >>> Thanks for reviewing the KIP.
>> >>>
>> >>>> I didn't follow the addition of a new method to the
>> QueryableStoreType
>> >>>> interface. Can you elaborate why this is necessary to support the new
>> >>>> exception types?
>> >>>
>> >>> To support the new exception types, I would check stream state in the
>> >>> following classes:
>> >>> - CompositeReadOnlyKeyValueStore class
>> >>> - CompositeReadOnlySessionStore class
>> >>> - CompositeReadOnlyWindowStore class
>> >>> - DelegatingPeekingKeyValueIterator class
>> >>>
>> >>> It is also necessary to keep backward compatibility. So I plan passing
>> >>> stream
>> >>> instance to QueryableStoreType instance during KafkaStreams#store()
>> >>> invoked.
>> >>> It looks a most simple way, I think.
>> >>>
>> >>> It is why I add a new method to the QueryableStoreType interface. I
>> can
>> >>> understand
>> >>> that we should try to avoid adding the public api method. However, at
>> the
>> >>> moment
>> >>> I have no better ideas.
>> >>>
>> >>> Any thoughts?
>> >>>
>> >>>
>> >>>> Also, looking over your KIP again, it seems valuable to introduce
>> >>>> "retriable store exception" and "fatal store exception" marker
>> interfaces
>> >>>> that the various exceptions can mix in. It would be nice from a
>> usability
>> >>>> perspective to be able to just log and retry on any "retriable"
>> exception
>> >>>> and log and shutdown on any fatal exception.
>> >>>
>> >>> I agree that this is valuable to the user.
>> >>> I'll update the KIP.
>> >>>
>> >>>
>> >>> Thanks
>> >>>
>> >>>
>> >>> ---
>> >>> Vito
>> >>>
>> >>>
>> >>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io>
>> wrote:
>> >>>
>> >>>> Hi Vito,
>> >>>>
>> >>>> I'm glad to hear you're well again!
>> >>>>
>> >>>> I didn't follow the addition of a new method to the
>> QueryableStoreType
>> >>>> interface. Can you elaborate why this is necessary to support the new
>> >>>> exception types?
>> >>>>
>> >>>> Also, looking over your KIP again, it seems valuable to introduce
>> >>>> "retriable store exception" and "fatal store exception" marker
>> interfaces
>> >>>> that the various exceptions can mix in. It would be nice from a
>> usability
>> >>>> perspective to be able to just log and retry on any "retriable"
>> exception
>> >>>> and log and shutdown on any fatal exception.
>> >>>>
>> >>>> Thanks,
>> >>>> -John
>> >>>>
>> >>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com>
>> wrote:
>> >>>>
>> >>>>> Thanks for the explanation, that makes sense.
>> >>>>>
>> >>>>>
>> >>>>> Guozhang
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
>> matthias@confluent.io
>> >>>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> The scenario I had I mind was, that KS is started in one thread
>> while
>> >>>> a
>> >>>>>> second thread has a reference to the object to issue queries.
>> >>>>>>
>> >>>>>> If a query is issue before the "main thread" started KS, and the
>> >>>> "query
>> >>>>>> thread" knows that it will eventually get started, it can retry. On
>> >>>> the
>> >>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>> >>>> impossible
>> >>>>>> to issue any query against it now or in the future and thus the
>> error
>> >>>> is
>> >>>>>> not retryable.
>> >>>>>>
>> >>>>>>
>> >>>>>> -Matthias
>> >>>>>>
>> >>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
>> >>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
>> >>>>>>> StreamThreadNotRunning, because I think users' handling logic for
>> >>>> the
>> >>>>>> third
>> >>>>>>> case would be likely the same as the second. Do you have some
>> >>>> scenarios
>> >>>>>>> where users may want to handle them differently?
>> >>>>>>>
>> >>>>>>> Guozhang
>> >>>>>>>
>> >>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>> >>>>> matthias@confluent.io>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Sorry to hear! Get well soon!
>> >>>>>>>>
>> >>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free to
>> >>>> pick
>> >>>>> it
>> >>>>>>>> up again when you find time.
>> >>>>>>>>
>> >>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable
>> error?
>> >>>>>>>>>>
>> >>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a
>> retryable
>> >>>>>> error.
>> >>>>>>>>>>
>> >>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>> >>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>> >>>>>>>> RUNNING. The
>> >>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>> >>>>>>>>
>> >>>>>>>> I see. If this is the intention, than I would suggest to have two
>> >>>> (or
>> >>>>>>>> maybe three) different exceptions:
>> >>>>>>>>
>> >>>>>>>> - StreamThreadRebalancingException (retryable)
>> >>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
>> >>>>>>>> PENDING_SHUTDOWN or DEAD
>> >>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
>> >>>>>>>>
>> >>>>>>>> The last one is tricky and could also be merged into one of the
>> >>>> first
>> >>>>>>>> two, depending if you want to argue that it's retryable or not.
>> >>>> (Just
>> >>>>>>>> food for though -- not sure what others think.)
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> -Matthias
>> >>>>>>>>
>> >>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
>> >>>>>>>>> Matthias,
>> >>>>>>>>>
>> >>>>>>>>> Thank you for your assistance.
>> >>>>>>>>>
>> >>>>>>>>>> what is the status of this KIP?
>> >>>>>>>>>
>> >>>>>>>>> Unfortunately, there is no further progress.
>> >>>>>>>>> About seven weeks ago, I was injured in sports. I had a broken
>> >>>> wrist
>> >>>>> on
>> >>>>>>>>> my left wrist.
>> >>>>>>>>> Many jobs are affected, including this KIP and implementation.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>> >>>> do
>> >>>>> we
>> >>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>> >>>> really
>> >>>>>> need
>> >>>>>>>>>> them? Can't we just throw the correct exception directly
>> instead
>> >>>> of
>> >>>>>>>>>> wrapping it later?
>> >>>>>>>>>
>> >>>>>>>>> I think you may be right. As I say in the previous:
>> >>>>>>>>> "The original idea is that we can distinguish different state
>> >>>> store
>> >>>>>>>>> exception for different handling. But to be honest, I am not
>> quite
>> >>>>> sure
>> >>>>>>>>> this is necessary. Maybe have some change during
>> implementation."
>> >>>>>>>>>
>> >>>>>>>>> During the implementation, I also feel we maybe not need wrapper
>> >>>> it.
>> >>>>>>>>> We can just throw the correctly directly.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>> >>>>>>>>>
>> >>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
>> >>>>> error.
>> >>>>>>>>>
>> >>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>> >>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>> >>>>>> RUNNING.
>> >>>>>>>> The
>> >>>>>>>>> user can retry until KafkaStream state is RUNNING.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>> semantics
>> >>>> is
>> >>>>>>>>> unclear to me atm.
>> >>>>>>>>>
>> >>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>> >>>>> retryable
>> >>>>>>>>> error?
>> >>>>>>>>>
>> >>>>>>>>> These two comments will be answered in another mail.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> ---
>> >>>>>>>>> Vito
>> >>>>>>>>>
>> >>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>> >>>>>> matthias@confluent.io>
>> >>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Vito,
>> >>>>>>>>>>
>> >>>>>>>>>> what is the status of this KIP?
>> >>>>>>>>>>
>> >>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>> >>>> do
>> >>>>> we
>> >>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>> >>>> really
>> >>>>>> need
>> >>>>>>>>>> them? Can't we just throw the correct exception directly
>> instead
>> >>>> of
>> >>>>>>>>>> wrapping it later?
>> >>>>>>>>>>
>> >>>>>>>>>> When would we throw an `StateStoreEmptyException`? The
>> semantics
>> >>>> is
>> >>>>>>>>>> unclear to me atm.
>> >>>>>>>>>>
>> >>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>> >>>>>>>>>>
>> >>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>> >>>>> retryable
>> >>>>>>>>>> error?
>> >>>>>>>>>>
>> >>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
>> key,
>> >>>>> long
>> >>>>>>>>>> time); that should be added
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Overall I like the KIP but some details are still unclear.
>> Maybe
>> >>>> it
>> >>>>>>>>>> might help if you open an PR in parallel?
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> -Matthias
>> >>>>>>>>>>
>> >>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>> >>>>>>>>>>> Hi, Guozhang,
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks for the comment!
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi, Bill,
>> >>>>>>>>>>>
>> >>>>>>>>>>> I'll try to make some update to make the KIP better.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks for the comment!
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ---
>> >>>>>>>>>>> Vito
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
>> bbejeck@gmail.com
>> >>>>>
>> >>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi Vito,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> At this point, the only thing I would change is possibly
>> >>>> removing
>> >>>>>> the
>> >>>>>>>>>>>> listing of all methods called by the user and the listing of
>> >>>> all
>> >>>>>> store
>> >>>>>>>>>>>> types and focus on what states result in which exceptions
>> >>>> thrown
>> >>>>> to
>> >>>>>>>> the
>> >>>>>>>>>>>> user.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>> Bill
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>> >>>>> wangguoz@gmail.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks for the KIP Vito!
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I made a pass over the wiki and it looks great to me. I'm +1
>> >>>> on
>> >>>>> the
>> >>>>>>>>>> KIP.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> About the base class InvalidStateStoreException itself, I'd
>> >>>>>> actually
>> >>>>>>>>>>>>> suggest we do not deprecate it but still expose it as part
>> of
>> >>>> the
>> >>>>>>>>>> public
>> >>>>>>>>>>>>> API, for people who do not want to handle these cases
>> >>>> differently
>> >>>>>> (if
>> >>>>>>>>>> we
>> >>>>>>>>>>>>> deprecate it then we are enforcing them to capture all three
>> >>>>>>>> exceptions
>> >>>>>>>>>>>>> one-by-one).
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Guozhang
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>> >>>> john@confluent.io
>> >>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Hi Vito,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks for the KIP!
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I think it's much nicer to give callers different
>> exceptions
>> >>>> to
>> >>>>>> tell
>> >>>>>>>>>>>> them
>> >>>>>>>>>>>>>> whether the state store got migrated, whether it's still
>> >>>>>>>> initializing,
>> >>>>>>>>>>>> or
>> >>>>>>>>>>>>>> whether there's some unrecoverable error.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
>> >>>>>> non-user-facing
>> >>>>>>>>>>>>> details
>> >>>>>>>>>>>>>> such as what exceptions we will throw internally. The KIP
>> is
>> >>>>>>>> primarily
>> >>>>>>>>>>>> to
>> >>>>>>>>>>>>>> discuss public interface changes.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> You might consider simply removing all the internal details
>> >>>> from
>> >>>>>> the
>> >>>>>>>>>>>> KIP,
>> >>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
>> >>>> smaller
>> >>>>>> and
>> >>>>>>>>>>>>> easier
>> >>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
>> >>>> internal
>> >>>>>>>>>> details
>> >>>>>>>>>>>>>> when it comes to implementation.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> I like your decision to have your refined exceptions extend
>> >>>>>>>>>>>>>> InvalidStateStoreException to ensure backward
>> compatibility.
>> >>>>> Since
>> >>>>>>>> we
>> >>>>>>>>>>>>> want
>> >>>>>>>>>>>>>> to encourage callers to catch the more specific exceptions,
>> >>>> and
>> >>>>> we
>> >>>>>>>>>>>> don't
>> >>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
>> >>>> anymore,
>> >>>>> you
>> >>>>>>>>>>>> might
>> >>>>>>>>>>>>>> consider adding the @Deprecated annotation to
>> >>>>>>>>>>>> InvalidStateStoreException.
>> >>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
>> >>>>> exception
>> >>>>>>>> and
>> >>>>>>>>>>>>> open
>> >>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
>> >>>> entirely
>> >>>>>> in a
>> >>>>>>>>>>>>> future
>> >>>>>>>>>>>>>> release.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>> -John
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>> >>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Can you update the KIP to make those suggestions explicit?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>> >>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Thanks for the feedback!
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>> not.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Got it!
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>> >>>>>>>>>>>> `StateStoreFailException`
>> >>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used?
>> Do
>> >>>>> you
>> >>>>>>>>>>>>> suggest
>> >>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>> >>>>> anymore,
>> >>>>>>>>>>>> but
>> >>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>> understanding).
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
>> >>>> thrown
>> >>>>>> at
>> >>>>>>>>>>>> all
>> >>>>>>>>>>>>>>>> anymore,
>> >>>>>>>>>>>>>>>> but only new sub-classes.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Not sure what this sentence means:
>> >>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>> >>>> exception
>> >>>>>>>>>>>>> finally.
>> >>>>>>>>>>>>>>>>> Can you elaborate?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
>> will
>> >>>>>> throw
>> >>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>> >>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
>> >>>>>>>>>>>>>>>> `StateStoreRetryableException` or
>> `StateStoreFailException`
>> >>>>>> during
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>> exceptions".
>> >>>>> It's
>> >>>>>>>>>>>>> unclear
>> >>>>>>>>>>>>>>>> to me atm why they are introduced.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>> >>>>>> distinguish
>> >>>>>>>>>>>>>> between
>> >>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
>> >>>>>>>>>>>>>>>> The original idea is that we can distinguish different
>> >>>> state
>> >>>>>> store
>> >>>>>>>>>>>>>>>> exception for
>> >>>>>>>>>>>>>>>> different handling.
>> >>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is necessary.
>> >>>> Maybe
>> >>>>>>>> have
>> >>>>>>>>>>>>>>>> some change during implementation.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Does it make sense?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>> >>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Thanks for the update Vito!
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
>> not.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>> >>>>>>>>>>>> `StateStoreFailException`
>> >>>>>>>>>>>>> or
>> >>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used?
>> Do
>> >>>>> you
>> >>>>>>>>>>>>> suggest
>> >>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>> >>>>> anymore,
>> >>>>>>>>>>>> but
>> >>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>> the new sub-classes (just to get a better
>> understanding).
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Not sure what this sentence means:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>> >>>> exception
>> >>>>>>>>>>>>> finally.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Can you elaborate?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal
>> exceptions".
>> >>>>> It's
>> >>>>>>>>>>>>> unclear
>> >>>>>>>>>>>>>>>>> to me atm why they are introduced.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Thanks for the review.
>> >>>>>>>>>>>>>>>>>> I reply separately in the following sections.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>> >>>>>>>>>>>>>> matthias@confluent.io
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>> >>>> pause...
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
>> code.
>> >>>>> It's
>> >>>>>>>>>>>>> useful
>> >>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>> understand what user facing interfaces are affected.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
>> >>>> `Changes
>> >>>>> in
>> >>>>>>>>>>>> call
>> >>>>>>>>>>>>>>>>> trace`.
>> >>>>>>>>>>>>>>>>>> Do you think it should be removed?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP, the
>> >>>> main
>> >>>>>>>>>>>> change
>> >>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
>> >>>>>>>>>>>> `InvalidStateStoreException`.
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important
>> things.
>> >>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions extend
>> >>>> from
>> >>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> Some questions:
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
>> >>>> `InvalidStateStoreException`
>> >>>>> be
>> >>>>>>>>>>>> used
>> >>>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>>>>> this purpose?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> Does this question miss some word?
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> - What the superclass of
>> >>>> `StateStoreStreamThreadNotRunni
>> >>>>>>>>>>>>>> ngException`
>> >>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>> >>>>>>>>>>>>>>>>> `StateStoreFailException`
>> >>>>>>>>>>>>>>>>>>> ?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable
>> exception
>> >>>> ?
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
>> modify
>> >>>>> some
>> >>>>>>>>>>>> code
>> >>>>>>>>>>>>>> in
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>> recent period and modify the KIP.
>> >>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
>> >>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>> >>>>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the state
>> >>>> both
>> >>>>>> of
>> >>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread
>> >>>> and
>> >>>>>> state
>> >>>>>>>>>>>>>> store,
>> >>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>> >>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Vito,
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is
>> not
>> >>>>>> hosted
>> >>>>>>>>>>>> an
>> >>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
>> >>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>> >>>> state
>> >>>>>>>>>>>>> rebalance
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
>> >>>> store.
>> >>>>>> For
>> >>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>> second
>> >>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
>> finished.
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>> >>>> PENDING_SHUTDOWN, or
>> >>>>>>>>>>>>>>> NOT_RUNNING,
>> >>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>> >>>>> rediscover
>> >>>>>> or
>> >>>>>>>>>>>>>> retry.
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
>> >>>> user
>> >>>>> can
>> >>>>>>>>>>>> just
>> >>>>>>>>>>>>>>> plain
>> >>>>>>>>>>>>>>>>>>>>>>> retry.
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN
>> or
>> >>>>>>>>>>>>> NOT_RUNNING,
>> >>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>>>>>> means
>> >>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the
>> user
>> >>>>> needs
>> >>>>>>>> to
>> >>>>>>>>>>>>>>>>>>> rediscover
>> >>>>>>>>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>> store.
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax
>> <
>> >>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>> >>>> idea of
>> >>>>>> the
>> >>>>>>>>>>>>> Jira
>> >>>>>>>>>>>>>>>>> was,
>> >>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
>> "recovery"
>> >>>>>>>>>>>>> strategies
>> >>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>> user.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated,
>> a
>> >>>>> user
>> >>>>>>>>>>>> just
>> >>>>>>>>>>>>>> need
>> >>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>> wait
>> >>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other
>> hand,
>> >>>> if
>> >>>>> a
>> >>>>>>>>>>>> store
>> >>>>>>>>>>>>>> go
>> >>>>>>>>>>>>>>>>>>>>> migrated
>> >>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover
>> >>>> the
>> >>>>>> store
>> >>>>>>>>>>>>>>> instead
>> >>>>>>>>>>>>>>>>>>> of a
>> >>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>> >>>> talks
>> >>>>>> about
>> >>>>>>>>>>>>> this
>> >>>>>>>>>>>>>>>>> ideas
>> >>>>>>>>>>>>>>>>>>>>> and
>> >>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
>> figure
>> >>>>> out
>> >>>>>>>>>>>> first
>> >>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>>>> what
>> >>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>> >>>> users, if
>> >>>>>> we
>> >>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>> wants
>> >>>>>>>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably
>> we
>> >>>>>> should
>> >>>>>>>>>>>>>> rename
>> >>>>>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>> >>>>>>>>>>>>>>> StateStoreClosedException
>> >>>>>>>>>>>>>>>>>>>>>>> classes.
>> >>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
>> >>>>>> possible
>> >>>>>>>>>>>>>>> internal
>> >>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the
>> call
>> >>>>>> trace,
>> >>>>>>>>>>>> and
>> >>>>>>>>>>>>>>> which
>> >>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others,
>> and
>> >>>>> which
>> >>>>>>>>>>>> ones
>> >>>>>>>>>>>>>>>>> should
>> >>>>>>>>>>>>>>>>>>> be
>> >>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>> >>>> should
>> >>>>> not
>> >>>>>>>> be
>> >>>>>>>>>>>>>>> wrapped
>> >>>>>>>>>>>>>>>>>>> at
>> >>>>>>>>>>>>>>>>>>>>>>> all
>> >>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>> >>>>>>>>>>>>>>> vito@is-land.com.tw>
>> >>>>>>>>>>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>> >>>>>> confluence/display/KAFKA/KIP-
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> 216%3A+IQ+should+throw+different+exceptions+for+
>> >>>>>>>>>>>>>>>>> different+errors
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> --
>> >>>>>>>>>>>>> -- Guozhang
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> -- Guozhang
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Vito Jeng <vi...@is-land.com.tw>.
Hi, Matthias,
Sorry for the late reply.
> I am wondering what the semantic impact/change is, if we introduce
> `RetryableStateStoreException` and `FatalStateStoreException` that both
> inherit from it. While I like the introduction of both from a high level
> point of view, I just want to make sure it's semantically sound and
> backward compatible. Atm, I think it's fine, but I want to point it out
> such that everybody can think about this, too, so we can verify that
> it's a natural evolving API change.
Thank you for pointing this out. This's really important for public API.
Just when I was replying to you, I found that KIP needs some modify.
I will fix it ASAP, and then let's continue the discussion.
---
Vito
On Wed, Nov 7, 2018 at 7:06 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Hey Vito,
>
> I saw that you updated your PR, but did not reply to my last comments.
> Any thoughts?
>
>
> -Matthias
>
> On 10/19/18 10:34 AM, Matthias J. Sax wrote:
> > Glad to have you back Vito :)
> >
> > Some follow up thoughts:
> >
> > - the current `InvalidStateStoreException` is documents as being
> > sometimes retry-able. From the JavaDocs:
> >
> >> These exceptions may be transient [...] Hence, it is valid to backoff
> and retry when handling this exception.
> >
> > I am wondering what the semantic impact/change is, if we introduce
> > `RetryableStateStoreException` and `FatalStateStoreException` that both
> > inherit from it. While I like the introduction of both from a high level
> > point of view, I just want to make sure it's semantically sound and
> > backward compatible. Atm, I think it's fine, but I want to point it out
> > such that everybody can think about this, too, so we can verify that
> > it's a natural evolving API change.
> >
> > - StateStoreClosedException:
> >
> >> will be wrapped to StateStoreMigratedException or
> StateStoreNotAvailableException later.
> >
> > Can you clarify the cases (ie, when will it be wrapped with the one or
> > the other)?
> >
> > - StateStoreIsEmptyException:
> >
> > I don't understand the semantic of this exception. Maybe it's a naming
> > issue?
> >
> >> will be wrapped to StateStoreMigratedException or
> StateStoreNotAvailableException later.
> >
> > Also, can you clarify the cases (ie, when will it be wrapped with the
> > one or the other)?
> >
> >
> > I am also wondering, if we should introduce a fatal exception
> > `UnkownStateStoreException` to tell users that they passed in an unknown
> > store name?
> >
> >
> >
> > -Matthias
> >
> >
> >
> > On 10/17/18 8:14 PM, vito jeng wrote:
> >> Just open a PR for further discussion:
> >> https://github.com/apache/kafka/pull/5814
> >>
> >> Any suggestion is welcome.
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >>
> >> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw> wrote:
> >>
> >>> Hi John,
> >>>
> >>> Thanks for reviewing the KIP.
> >>>
> >>>> I didn't follow the addition of a new method to the QueryableStoreType
> >>>> interface. Can you elaborate why this is necessary to support the new
> >>>> exception types?
> >>>
> >>> To support the new exception types, I would check stream state in the
> >>> following classes:
> >>> - CompositeReadOnlyKeyValueStore class
> >>> - CompositeReadOnlySessionStore class
> >>> - CompositeReadOnlyWindowStore class
> >>> - DelegatingPeekingKeyValueIterator class
> >>>
> >>> It is also necessary to keep backward compatibility. So I plan passing
> >>> stream
> >>> instance to QueryableStoreType instance during KafkaStreams#store()
> >>> invoked.
> >>> It looks a most simple way, I think.
> >>>
> >>> It is why I add a new method to the QueryableStoreType interface. I can
> >>> understand
> >>> that we should try to avoid adding the public api method. However, at
> the
> >>> moment
> >>> I have no better ideas.
> >>>
> >>> Any thoughts?
> >>>
> >>>
> >>>> Also, looking over your KIP again, it seems valuable to introduce
> >>>> "retriable store exception" and "fatal store exception" marker
> interfaces
> >>>> that the various exceptions can mix in. It would be nice from a
> usability
> >>>> perspective to be able to just log and retry on any "retriable"
> exception
> >>>> and log and shutdown on any fatal exception.
> >>>
> >>> I agree that this is valuable to the user.
> >>> I'll update the KIP.
> >>>
> >>>
> >>> Thanks
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>>
> >>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io> wrote:
> >>>
> >>>> Hi Vito,
> >>>>
> >>>> I'm glad to hear you're well again!
> >>>>
> >>>> I didn't follow the addition of a new method to the QueryableStoreType
> >>>> interface. Can you elaborate why this is necessary to support the new
> >>>> exception types?
> >>>>
> >>>> Also, looking over your KIP again, it seems valuable to introduce
> >>>> "retriable store exception" and "fatal store exception" marker
> interfaces
> >>>> that the various exceptions can mix in. It would be nice from a
> usability
> >>>> perspective to be able to just log and retry on any "retriable"
> exception
> >>>> and log and shutdown on any fatal exception.
> >>>>
> >>>> Thanks,
> >>>> -John
> >>>>
> >>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com>
> wrote:
> >>>>
> >>>>> Thanks for the explanation, that makes sense.
> >>>>>
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <
> matthias@confluent.io
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> The scenario I had I mind was, that KS is started in one thread
> while
> >>>> a
> >>>>>> second thread has a reference to the object to issue queries.
> >>>>>>
> >>>>>> If a query is issue before the "main thread" started KS, and the
> >>>> "query
> >>>>>> thread" knows that it will eventually get started, it can retry. On
> >>>> the
> >>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
> >>>> impossible
> >>>>>> to issue any query against it now or in the future and thus the
> error
> >>>> is
> >>>>>> not retryable.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
> >>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
> >>>>>>> StreamThreadNotRunning, because I think users' handling logic for
> >>>> the
> >>>>>> third
> >>>>>>> case would be likely the same as the second. Do you have some
> >>>> scenarios
> >>>>>>> where users may want to handle them differently?
> >>>>>>>
> >>>>>>> Guozhang
> >>>>>>>
> >>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
> >>>>> matthias@confluent.io>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Sorry to hear! Get well soon!
> >>>>>>>>
> >>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free to
> >>>> pick
> >>>>> it
> >>>>>>>> up again when you find time.
> >>>>>>>>
> >>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
> >>>>>>>>>>
> >>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
> >>>>>> error.
> >>>>>>>>>>
> >>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
> >>>>>>>> RUNNING. The
> >>>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>
> >>>>>>>> I see. If this is the intention, than I would suggest to have two
> >>>> (or
> >>>>>>>> maybe three) different exceptions:
> >>>>>>>>
> >>>>>>>> - StreamThreadRebalancingException (retryable)
> >>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
> >>>>>>>> PENDING_SHUTDOWN or DEAD
> >>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
> >>>>>>>>
> >>>>>>>> The last one is tricky and could also be merged into one of the
> >>>> first
> >>>>>>>> two, depending if you want to argue that it's retryable or not.
> >>>> (Just
> >>>>>>>> food for though -- not sure what others think.)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
> >>>>>>>>> Matthias,
> >>>>>>>>>
> >>>>>>>>> Thank you for your assistance.
> >>>>>>>>>
> >>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>
> >>>>>>>>> Unfortunately, there is no further progress.
> >>>>>>>>> About seven weeks ago, I was injured in sports. I had a broken
> >>>> wrist
> >>>>> on
> >>>>>>>>> my left wrist.
> >>>>>>>>> Many jobs are affected, including this KIP and implementation.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
> >>>> do
> >>>>> we
> >>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
> >>>> really
> >>>>>> need
> >>>>>>>>>> them? Can't we just throw the correct exception directly instead
> >>>> of
> >>>>>>>>>> wrapping it later?
> >>>>>>>>>
> >>>>>>>>> I think you may be right. As I say in the previous:
> >>>>>>>>> "The original idea is that we can distinguish different state
> >>>> store
> >>>>>>>>> exception for different handling. But to be honest, I am not
> quite
> >>>>> sure
> >>>>>>>>> this is necessary. Maybe have some change during implementation."
> >>>>>>>>>
> >>>>>>>>> During the implementation, I also feel we maybe not need wrapper
> >>>> it.
> >>>>>>>>> We can just throw the correctly directly.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
> >>>>>>>>>
> >>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
> >>>>> error.
> >>>>>>>>>
> >>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
> >>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
> >>>>>> RUNNING.
> >>>>>>>> The
> >>>>>>>>> user can retry until KafkaStream state is RUNNING.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> When would we throw an `StateStoreEmptyException`? The semantics
> >>>> is
> >>>>>>>>> unclear to me atm.
> >>>>>>>>>
> >>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
> >>>>> retryable
> >>>>>>>>> error?
> >>>>>>>>>
> >>>>>>>>> These two comments will be answered in another mail.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>> Vito
> >>>>>>>>>
> >>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
> >>>>>> matthias@confluent.io>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Vito,
> >>>>>>>>>>
> >>>>>>>>>> what is the status of this KIP?
> >>>>>>>>>>
> >>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
> >>>> do
> >>>>> we
> >>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
> >>>> really
> >>>>>> need
> >>>>>>>>>> them? Can't we just throw the correct exception directly instead
> >>>> of
> >>>>>>>>>> wrapping it later?
> >>>>>>>>>>
> >>>>>>>>>> When would we throw an `StateStoreEmptyException`? The semantics
> >>>> is
> >>>>>>>>>> unclear to me atm.
> >>>>>>>>>>
> >>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
> >>>>>>>>>>
> >>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
> >>>>> retryable
> >>>>>>>>>> error?
> >>>>>>>>>>
> >>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K
> key,
> >>>>> long
> >>>>>>>>>> time); that should be added
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Overall I like the KIP but some details are still unclear. Maybe
> >>>> it
> >>>>>>>>>> might help if you open an PR in parallel?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
> >>>>>>>>>>> Hi, Guozhang,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Hi, Bill,
> >>>>>>>>>>>
> >>>>>>>>>>> I'll try to make some update to make the KIP better.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the comment!
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ---
> >>>>>>>>>>> Vito
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <
> bbejeck@gmail.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
> >>>>>>>>>>>>
> >>>>>>>>>>>> At this point, the only thing I would change is possibly
> >>>> removing
> >>>>>> the
> >>>>>>>>>>>> listing of all methods called by the user and the listing of
> >>>> all
> >>>>>> store
> >>>>>>>>>>>> types and focus on what states result in which exceptions
> >>>> thrown
> >>>>> to
> >>>>>>>> the
> >>>>>>>>>>>> user.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Bill
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
> >>>>> wangguoz@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I made a pass over the wiki and it looks great to me. I'm +1
> >>>> on
> >>>>> the
> >>>>>>>>>> KIP.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> About the base class InvalidStateStoreException itself, I'd
> >>>>>> actually
> >>>>>>>>>>>>> suggest we do not deprecate it but still expose it as part of
> >>>> the
> >>>>>>>>>> public
> >>>>>>>>>>>>> API, for people who do not want to handle these cases
> >>>> differently
> >>>>>> (if
> >>>>>>>>>> we
> >>>>>>>>>>>>> deprecate it then we are enforcing them to capture all three
> >>>>>>>> exceptions
> >>>>>>>>>>>>> one-by-one).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
> >>>> john@confluent.io
> >>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Vito,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks for the KIP!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think it's much nicer to give callers different exceptions
> >>>> to
> >>>>>> tell
> >>>>>>>>>>>> them
> >>>>>>>>>>>>>> whether the state store got migrated, whether it's still
> >>>>>>>> initializing,
> >>>>>>>>>>>> or
> >>>>>>>>>>>>>> whether there's some unrecoverable error.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
> >>>>>> non-user-facing
> >>>>>>>>>>>>> details
> >>>>>>>>>>>>>> such as what exceptions we will throw internally. The KIP is
> >>>>>>>> primarily
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>> discuss public interface changes.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> You might consider simply removing all the internal details
> >>>> from
> >>>>>> the
> >>>>>>>>>>>> KIP,
> >>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
> >>>> smaller
> >>>>>> and
> >>>>>>>>>>>>> easier
> >>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
> >>>> internal
> >>>>>>>>>> details
> >>>>>>>>>>>>>> when it comes to implementation.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I like your decision to have your refined exceptions extend
> >>>>>>>>>>>>>> InvalidStateStoreException to ensure backward compatibility.
> >>>>> Since
> >>>>>>>> we
> >>>>>>>>>>>>> want
> >>>>>>>>>>>>>> to encourage callers to catch the more specific exceptions,
> >>>> and
> >>>>> we
> >>>>>>>>>>>> don't
> >>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
> >>>> anymore,
> >>>>> you
> >>>>>>>>>>>> might
> >>>>>>>>>>>>>> consider adding the @Deprecated annotation to
> >>>>>>>>>>>> InvalidStateStoreException.
> >>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
> >>>>> exception
> >>>>>>>> and
> >>>>>>>>>>>>> open
> >>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
> >>>> entirely
> >>>>>> in a
> >>>>>>>>>>>>> future
> >>>>>>>>>>>>>> release.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Can you update the KIP to make those suggestions explicit?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the feedback!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
> not.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Got it!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used?
> Do
> >>>>> you
> >>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> >>>>> anymore,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
> >>>> thrown
> >>>>>> at
> >>>>>>>>>>>> all
> >>>>>>>>>>>>>>>> anymore,
> >>>>>>>>>>>>>>>> but only new sub-classes.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
> >>>> exception
> >>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()`
> will
> >>>>>> throw
> >>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
> >>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
> >>>>>>>>>>>>>>>> `StateStoreRetryableException` or
> `StateStoreFailException`
> >>>>>> during
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
> >>>>> It's
> >>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
> >>>>>> distinguish
> >>>>>>>>>>>>>> between
> >>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
> >>>>>>>>>>>>>>>> The original idea is that we can distinguish different
> >>>> state
> >>>>>> store
> >>>>>>>>>>>>>>>> exception for
> >>>>>>>>>>>>>>>> different handling.
> >>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is necessary.
> >>>> Maybe
> >>>>>>>> have
> >>>>>>>>>>>>>>>> some change during implementation.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Does it make sense?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the update Vito!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or
> not.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used?
> Do
> >>>>> you
> >>>>>>>>>>>>> suggest
> >>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> >>>>> anymore,
> >>>>>>>>>>>> but
> >>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
> >>>> exception
> >>>>>>>>>>>>> finally.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
> >>>>> It's
> >>>>>>>>>>>>> unclear
> >>>>>>>>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for the review.
> >>>>>>>>>>>>>>>>>> I reply separately in the following sections.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>>>>>>>>>>>> matthias@confluent.io
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
> >>>> pause...
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the
> code.
> >>>>> It's
> >>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> understand what user facing interfaces are affected.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
> >>>> `Changes
> >>>>> in
> >>>>>>>>>>>> call
> >>>>>>>>>>>>>>>>> trace`.
> >>>>>>>>>>>>>>>>>> Do you think it should be removed?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP, the
> >>>> main
> >>>>>>>>>>>> change
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
> >>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
> >>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions extend
> >>>> from
> >>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Some questions:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
> >>>> `InvalidStateStoreException`
> >>>>> be
> >>>>>>>>>>>> used
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>> this purpose?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Does this question miss some word?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> - What the superclass of
> >>>> `StateStoreStreamThreadNotRunni
> >>>>>>>>>>>>>> ngException`
> >>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>>>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception
> >>>> ?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to
> modify
> >>>>> some
> >>>>>>>>>>>> code
> >>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> recent period and modify the KIP.
> >>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the state
> >>>> both
> >>>>>> of
> >>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread
> >>>> and
> >>>>>> state
> >>>>>>>>>>>>>> store,
> >>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not
> >>>>>> hosted
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
> >>>> state
> >>>>>>>>>>>>> rebalance
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
> >>>> store.
> >>>>>> For
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> second
> >>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is
> finished.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
> >>>> PENDING_SHUTDOWN, or
> >>>>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
> >>>>> rediscover
> >>>>>> or
> >>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
> >>>> user
> >>>>> can
> >>>>>>>>>>>> just
> >>>>>>>>>>>>>>> plain
> >>>>>>>>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> >>>>>>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the user
> >>>>> needs
> >>>>>>>> to
> >>>>>>>>>>>>>>>>>>> rediscover
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
> >>>> idea of
> >>>>>> the
> >>>>>>>>>>>>> Jira
> >>>>>>>>>>>>>>>>> was,
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different
> "recovery"
> >>>>>>>>>>>>> strategies
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a
> >>>>> user
> >>>>>>>>>>>> just
> >>>>>>>>>>>>>> need
> >>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>> wait
> >>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand,
> >>>> if
> >>>>> a
> >>>>>>>>>>>> store
> >>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>>>> migrated
> >>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover
> >>>> the
> >>>>>> store
> >>>>>>>>>>>>>>> instead
> >>>>>>>>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
> >>>> talks
> >>>>>> about
> >>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>> ideas
> >>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to
> figure
> >>>>> out
> >>>>>>>>>>>> first
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
> >>>> users, if
> >>>>>> we
> >>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>> wants
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably
> we
> >>>>>> should
> >>>>>>>>>>>>>> rename
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>>>>>>>>>>>> StateStoreClosedException
> >>>>>>>>>>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
> >>>>>> possible
> >>>>>>>>>>>>>>> internal
> >>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call
> >>>>>> trace,
> >>>>>>>>>>>> and
> >>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and
> >>>>> which
> >>>>>>>>>>>> ones
> >>>>>>>>>>>>>>>>> should
> >>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
> >>>> should
> >>>>> not
> >>>>>>>> be
> >>>>>>>>>>>>>>> wrapped
> >>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> >>>>>>>>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
> >>>>>> confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>>>>>>>>>>>> different+errors
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> -- Guozhang
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Hey Vito,
I saw that you updated your PR, but did not reply to my last comments.
Any thoughts?
-Matthias
On 10/19/18 10:34 AM, Matthias J. Sax wrote:
> Glad to have you back Vito :)
>
> Some follow up thoughts:
>
> - the current `InvalidStateStoreException` is documents as being
> sometimes retry-able. From the JavaDocs:
>
>> These exceptions may be transient [...] Hence, it is valid to backoff and retry when handling this exception.
>
> I am wondering what the semantic impact/change is, if we introduce
> `RetryableStateStoreException` and `FatalStateStoreException` that both
> inherit from it. While I like the introduction of both from a high level
> point of view, I just want to make sure it's semantically sound and
> backward compatible. Atm, I think it's fine, but I want to point it out
> such that everybody can think about this, too, so we can verify that
> it's a natural evolving API change.
>
> - StateStoreClosedException:
>
>> will be wrapped to StateStoreMigratedException or StateStoreNotAvailableException later.
>
> Can you clarify the cases (ie, when will it be wrapped with the one or
> the other)?
>
> - StateStoreIsEmptyException:
>
> I don't understand the semantic of this exception. Maybe it's a naming
> issue?
>
>> will be wrapped to StateStoreMigratedException or StateStoreNotAvailableException later.
>
> Also, can you clarify the cases (ie, when will it be wrapped with the
> one or the other)?
>
>
> I am also wondering, if we should introduce a fatal exception
> `UnkownStateStoreException` to tell users that they passed in an unknown
> store name?
>
>
>
> -Matthias
>
>
>
> On 10/17/18 8:14 PM, vito jeng wrote:
>> Just open a PR for further discussion:
>> https://github.com/apache/kafka/pull/5814
>>
>> Any suggestion is welcome.
>> Thanks!
>>
>> ---
>> Vito
>>
>>
>> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw> wrote:
>>
>>> Hi John,
>>>
>>> Thanks for reviewing the KIP.
>>>
>>>> I didn't follow the addition of a new method to the QueryableStoreType
>>>> interface. Can you elaborate why this is necessary to support the new
>>>> exception types?
>>>
>>> To support the new exception types, I would check stream state in the
>>> following classes:
>>> - CompositeReadOnlyKeyValueStore class
>>> - CompositeReadOnlySessionStore class
>>> - CompositeReadOnlyWindowStore class
>>> - DelegatingPeekingKeyValueIterator class
>>>
>>> It is also necessary to keep backward compatibility. So I plan passing
>>> stream
>>> instance to QueryableStoreType instance during KafkaStreams#store()
>>> invoked.
>>> It looks a most simple way, I think.
>>>
>>> It is why I add a new method to the QueryableStoreType interface. I can
>>> understand
>>> that we should try to avoid adding the public api method. However, at the
>>> moment
>>> I have no better ideas.
>>>
>>> Any thoughts?
>>>
>>>
>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>> "retriable store exception" and "fatal store exception" marker interfaces
>>>> that the various exceptions can mix in. It would be nice from a usability
>>>> perspective to be able to just log and retry on any "retriable" exception
>>>> and log and shutdown on any fatal exception.
>>>
>>> I agree that this is valuable to the user.
>>> I'll update the KIP.
>>>
>>>
>>> Thanks
>>>
>>>
>>> ---
>>> Vito
>>>
>>>
>>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io> wrote:
>>>
>>>> Hi Vito,
>>>>
>>>> I'm glad to hear you're well again!
>>>>
>>>> I didn't follow the addition of a new method to the QueryableStoreType
>>>> interface. Can you elaborate why this is necessary to support the new
>>>> exception types?
>>>>
>>>> Also, looking over your KIP again, it seems valuable to introduce
>>>> "retriable store exception" and "fatal store exception" marker interfaces
>>>> that the various exceptions can mix in. It would be nice from a usability
>>>> perspective to be able to just log and retry on any "retriable" exception
>>>> and log and shutdown on any fatal exception.
>>>>
>>>> Thanks,
>>>> -John
>>>>
>>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com> wrote:
>>>>
>>>>> Thanks for the explanation, that makes sense.
>>>>>
>>>>>
>>>>> Guozhang
>>>>>
>>>>>
>>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <matthias@confluent.io
>>>>>
>>>>> wrote:
>>>>>
>>>>>> The scenario I had I mind was, that KS is started in one thread while
>>>> a
>>>>>> second thread has a reference to the object to issue queries.
>>>>>>
>>>>>> If a query is issue before the "main thread" started KS, and the
>>>> "query
>>>>>> thread" knows that it will eventually get started, it can retry. On
>>>> the
>>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>>>> impossible
>>>>>> to issue any query against it now or in the future and thus the error
>>>> is
>>>>>> not retryable.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
>>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
>>>>>>> StreamThreadNotRunning, because I think users' handling logic for
>>>> the
>>>>>> third
>>>>>>> case would be likely the same as the second. Do you have some
>>>> scenarios
>>>>>>> where users may want to handle them differently?
>>>>>>>
>>>>>>> Guozhang
>>>>>>>
>>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>>>>> matthias@confluent.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry to hear! Get well soon!
>>>>>>>>
>>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free to
>>>> pick
>>>>> it
>>>>>>>> up again when you find time.
>>>>>>>>
>>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>>>
>>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
>>>>>> error.
>>>>>>>>>>
>>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>>>>>>>> RUNNING. The
>>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>
>>>>>>>> I see. If this is the intention, than I would suggest to have two
>>>> (or
>>>>>>>> maybe three) different exceptions:
>>>>>>>>
>>>>>>>> - StreamThreadRebalancingException (retryable)
>>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
>>>>>>>> PENDING_SHUTDOWN or DEAD
>>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
>>>>>>>>
>>>>>>>> The last one is tricky and could also be merged into one of the
>>>> first
>>>>>>>> two, depending if you want to argue that it's retryable or not.
>>>> (Just
>>>>>>>> food for though -- not sure what others think.)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
>>>>>>>>> Matthias,
>>>>>>>>>
>>>>>>>>> Thank you for your assistance.
>>>>>>>>>
>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>
>>>>>>>>> Unfortunately, there is no further progress.
>>>>>>>>> About seven weeks ago, I was injured in sports. I had a broken
>>>> wrist
>>>>> on
>>>>>>>>> my left wrist.
>>>>>>>>> Many jobs are affected, including this KIP and implementation.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>>>> do
>>>>> we
>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>> really
>>>>>> need
>>>>>>>>>> them? Can't we just throw the correct exception directly instead
>>>> of
>>>>>>>>>> wrapping it later?
>>>>>>>>>
>>>>>>>>> I think you may be right. As I say in the previous:
>>>>>>>>> "The original idea is that we can distinguish different state
>>>> store
>>>>>>>>> exception for different handling. But to be honest, I am not quite
>>>>> sure
>>>>>>>>> this is necessary. Maybe have some change during implementation."
>>>>>>>>>
>>>>>>>>> During the implementation, I also feel we maybe not need wrapper
>>>> it.
>>>>>>>>> We can just throw the correctly directly.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>>
>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
>>>>> error.
>>>>>>>>>
>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>>>>>> RUNNING.
>>>>>>>> The
>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The semantics
>>>> is
>>>>>>>>> unclear to me atm.
>>>>>>>>>
>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>> retryable
>>>>>>>>> error?
>>>>>>>>>
>>>>>>>>> These two comments will be answered in another mail.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Vito
>>>>>>>>>
>>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>>>>>> matthias@confluent.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Vito,
>>>>>>>>>>
>>>>>>>>>> what is the status of this KIP?
>>>>>>>>>>
>>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>>>> do
>>>>> we
>>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>>> really
>>>>>> need
>>>>>>>>>> them? Can't we just throw the correct exception directly instead
>>>> of
>>>>>>>>>> wrapping it later?
>>>>>>>>>>
>>>>>>>>>> When would we throw an `StateStoreEmptyException`? The semantics
>>>> is
>>>>>>>>>> unclear to me atm.
>>>>>>>>>>
>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>>>
>>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>>> retryable
>>>>>>>>>> error?
>>>>>>>>>>
>>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key,
>>>>> long
>>>>>>>>>> time); that should be added
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Overall I like the KIP but some details are still unclear. Maybe
>>>> it
>>>>>>>>>> might help if you open an PR in parallel?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>>>>>>>>>>> Hi, Guozhang,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi, Bill,
>>>>>>>>>>>
>>>>>>>>>>> I'll try to make some update to make the KIP better.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> Vito
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bbejeck@gmail.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
>>>>>>>>>>>>
>>>>>>>>>>>> At this point, the only thing I would change is possibly
>>>> removing
>>>>>> the
>>>>>>>>>>>> listing of all methods called by the user and the listing of
>>>> all
>>>>>> store
>>>>>>>>>>>> types and focus on what states result in which exceptions
>>>> thrown
>>>>> to
>>>>>>>> the
>>>>>>>>>>>> user.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Bill
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>>>>> wangguoz@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I made a pass over the wiki and it looks great to me. I'm +1
>>>> on
>>>>> the
>>>>>>>>>> KIP.
>>>>>>>>>>>>>
>>>>>>>>>>>>> About the base class InvalidStateStoreException itself, I'd
>>>>>> actually
>>>>>>>>>>>>> suggest we do not deprecate it but still expose it as part of
>>>> the
>>>>>>>>>> public
>>>>>>>>>>>>> API, for people who do not want to handle these cases
>>>> differently
>>>>>> (if
>>>>>>>>>> we
>>>>>>>>>>>>> deprecate it then we are enforcing them to capture all three
>>>>>>>> exceptions
>>>>>>>>>>>>> one-by-one).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>>>> john@confluent.io
>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think it's much nicer to give callers different exceptions
>>>> to
>>>>>> tell
>>>>>>>>>>>> them
>>>>>>>>>>>>>> whether the state store got migrated, whether it's still
>>>>>>>> initializing,
>>>>>>>>>>>> or
>>>>>>>>>>>>>> whether there's some unrecoverable error.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
>>>>>> non-user-facing
>>>>>>>>>>>>> details
>>>>>>>>>>>>>> such as what exceptions we will throw internally. The KIP is
>>>>>>>> primarily
>>>>>>>>>>>> to
>>>>>>>>>>>>>> discuss public interface changes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You might consider simply removing all the internal details
>>>> from
>>>>>> the
>>>>>>>>>>>> KIP,
>>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
>>>> smaller
>>>>>> and
>>>>>>>>>>>>> easier
>>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
>>>> internal
>>>>>>>>>> details
>>>>>>>>>>>>>> when it comes to implementation.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I like your decision to have your refined exceptions extend
>>>>>>>>>>>>>> InvalidStateStoreException to ensure backward compatibility.
>>>>> Since
>>>>>>>> we
>>>>>>>>>>>>> want
>>>>>>>>>>>>>> to encourage callers to catch the more specific exceptions,
>>>> and
>>>>> we
>>>>>>>>>>>> don't
>>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
>>>> anymore,
>>>>> you
>>>>>>>>>>>> might
>>>>>>>>>>>>>> consider adding the @Deprecated annotation to
>>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
>>>>> exception
>>>>>>>> and
>>>>>>>>>>>>> open
>>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
>>>> entirely
>>>>>> in a
>>>>>>>>>>>>> future
>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> -John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you update the KIP to make those suggestions explicit?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Got it!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
>>>>> you
>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>>>>> anymore,
>>>>>>>>>>>> but
>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
>>>> thrown
>>>>>> at
>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>>> but only new sub-classes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>> exception
>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will
>>>>>> throw
>>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
>>>>>>>>>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException`
>>>>>> during
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
>>>>> It's
>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>>>>>> distinguish
>>>>>>>>>>>>>> between
>>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
>>>>>>>>>>>>>>>> The original idea is that we can distinguish different
>>>> state
>>>>>> store
>>>>>>>>>>>>>>>> exception for
>>>>>>>>>>>>>>>> different handling.
>>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is necessary.
>>>> Maybe
>>>>>>>> have
>>>>>>>>>>>>>>>> some change during implementation.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Does it make sense?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the update Vito!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
>>>>> you
>>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>>>>> anymore,
>>>>>>>>>>>> but
>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>>> exception
>>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
>>>>> It's
>>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the review.
>>>>>>>>>>>>>>>>>> I reply separately in the following sections.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>>>> pause...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the code.
>>>>> It's
>>>>>>>>>>>>> useful
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> understand what user facing interfaces are affected.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
>>>> `Changes
>>>>> in
>>>>>>>>>>>> call
>>>>>>>>>>>>>>>>> trace`.
>>>>>>>>>>>>>>>>>> Do you think it should be removed?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP, the
>>>> main
>>>>>>>>>>>> change
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
>>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions extend
>>>> from
>>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Some questions:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
>>>> `InvalidStateStoreException`
>>>>> be
>>>>>>>>>>>> used
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>> this purpose?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does this question miss some word?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - What the superclass of
>>>> `StateStoreStreamThreadNotRunni
>>>>>>>>>>>>>> ngException`
>>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception
>>>> ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to modify
>>>>> some
>>>>>>>>>>>> code
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> recent period and modify the KIP.
>>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the state
>>>> both
>>>>>> of
>>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread
>>>> and
>>>>>> state
>>>>>>>>>>>>>> store,
>>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not
>>>>>> hosted
>>>>>>>>>>>> an
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>>>> state
>>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
>>>> store.
>>>>>> For
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>>>> PENDING_SHUTDOWN, or
>>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>>>>> rediscover
>>>>>> or
>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
>>>> user
>>>>> can
>>>>>>>>>>>> just
>>>>>>>>>>>>>>> plain
>>>>>>>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the user
>>>>> needs
>>>>>>>> to
>>>>>>>>>>>>>>>>>>> rediscover
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>>>> idea of
>>>>>> the
>>>>>>>>>>>>> Jira
>>>>>>>>>>>>>>>>> was,
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
>>>>>>>>>>>>> strategies
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a
>>>>> user
>>>>>>>>>>>> just
>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand,
>>>> if
>>>>> a
>>>>>>>>>>>> store
>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover
>>>> the
>>>>>> store
>>>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>>>> talks
>>>>>> about
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>> ideas
>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure
>>>>> out
>>>>>>>>>>>> first
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>>>> users, if
>>>>>> we
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we
>>>>>> should
>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>>>>>>>>>>>> StateStoreClosedException
>>>>>>>>>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
>>>>>> possible
>>>>>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call
>>>>>> trace,
>>>>>>>>>>>> and
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and
>>>>> which
>>>>>>>>>>>> ones
>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>>>> should
>>>>> not
>>>>>>>> be
>>>>>>>>>>>>>>> wrapped
>>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>>>>>> confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>>>>>>>>>>>> different+errors
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -- Guozhang
>>>>>
>>>>
>>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Glad to have you back Vito :)
Some follow up thoughts:
- the current `InvalidStateStoreException` is documents as being
sometimes retry-able. From the JavaDocs:
> These exceptions may be transient [...] Hence, it is valid to backoff and retry when handling this exception.
I am wondering what the semantic impact/change is, if we introduce
`RetryableStateStoreException` and `FatalStateStoreException` that both
inherit from it. While I like the introduction of both from a high level
point of view, I just want to make sure it's semantically sound and
backward compatible. Atm, I think it's fine, but I want to point it out
such that everybody can think about this, too, so we can verify that
it's a natural evolving API change.
- StateStoreClosedException:
> will be wrapped to StateStoreMigratedException or StateStoreNotAvailableException later.
Can you clarify the cases (ie, when will it be wrapped with the one or
the other)?
- StateStoreIsEmptyException:
I don't understand the semantic of this exception. Maybe it's a naming
issue?
> will be wrapped to StateStoreMigratedException or StateStoreNotAvailableException later.
Also, can you clarify the cases (ie, when will it be wrapped with the
one or the other)?
I am also wondering, if we should introduce a fatal exception
`UnkownStateStoreException` to tell users that they passed in an unknown
store name?
-Matthias
On 10/17/18 8:14 PM, vito jeng wrote:
> Just open a PR for further discussion:
> https://github.com/apache/kafka/pull/5814
>
> Any suggestion is welcome.
> Thanks!
>
> ---
> Vito
>
>
> On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw> wrote:
>
>> Hi John,
>>
>> Thanks for reviewing the KIP.
>>
>>> I didn't follow the addition of a new method to the QueryableStoreType
>>> interface. Can you elaborate why this is necessary to support the new
>>> exception types?
>>
>> To support the new exception types, I would check stream state in the
>> following classes:
>> - CompositeReadOnlyKeyValueStore class
>> - CompositeReadOnlySessionStore class
>> - CompositeReadOnlyWindowStore class
>> - DelegatingPeekingKeyValueIterator class
>>
>> It is also necessary to keep backward compatibility. So I plan passing
>> stream
>> instance to QueryableStoreType instance during KafkaStreams#store()
>> invoked.
>> It looks a most simple way, I think.
>>
>> It is why I add a new method to the QueryableStoreType interface. I can
>> understand
>> that we should try to avoid adding the public api method. However, at the
>> moment
>> I have no better ideas.
>>
>> Any thoughts?
>>
>>
>>> Also, looking over your KIP again, it seems valuable to introduce
>>> "retriable store exception" and "fatal store exception" marker interfaces
>>> that the various exceptions can mix in. It would be nice from a usability
>>> perspective to be able to just log and retry on any "retriable" exception
>>> and log and shutdown on any fatal exception.
>>
>> I agree that this is valuable to the user.
>> I'll update the KIP.
>>
>>
>> Thanks
>>
>>
>> ---
>> Vito
>>
>>
>> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io> wrote:
>>
>>> Hi Vito,
>>>
>>> I'm glad to hear you're well again!
>>>
>>> I didn't follow the addition of a new method to the QueryableStoreType
>>> interface. Can you elaborate why this is necessary to support the new
>>> exception types?
>>>
>>> Also, looking over your KIP again, it seems valuable to introduce
>>> "retriable store exception" and "fatal store exception" marker interfaces
>>> that the various exceptions can mix in. It would be nice from a usability
>>> perspective to be able to just log and retry on any "retriable" exception
>>> and log and shutdown on any fatal exception.
>>>
>>> Thanks,
>>> -John
>>>
>>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com> wrote:
>>>
>>>> Thanks for the explanation, that makes sense.
>>>>
>>>>
>>>> Guozhang
>>>>
>>>>
>>>> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <matthias@confluent.io
>>>>
>>>> wrote:
>>>>
>>>>> The scenario I had I mind was, that KS is started in one thread while
>>> a
>>>>> second thread has a reference to the object to issue queries.
>>>>>
>>>>> If a query is issue before the "main thread" started KS, and the
>>> "query
>>>>> thread" knows that it will eventually get started, it can retry. On
>>> the
>>>>> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>>> impossible
>>>>> to issue any query against it now or in the future and thus the error
>>> is
>>>>> not retryable.
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 6/25/18 10:15 AM, Guozhang Wang wrote:
>>>>>> I'm wondering if StreamThreadNotStarted could be merged into
>>>>>> StreamThreadNotRunning, because I think users' handling logic for
>>> the
>>>>> third
>>>>>> case would be likely the same as the second. Do you have some
>>> scenarios
>>>>>> where users may want to handle them differently?
>>>>>>
>>>>>> Guozhang
>>>>>>
>>>>>> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>>>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry to hear! Get well soon!
>>>>>>>
>>>>>>> It's not a big deal if the KIP stalls a little bit. Feel free to
>>> pick
>>>> it
>>>>>>> up again when you find time.
>>>>>>>
>>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>>
>>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
>>>>> error.
>>>>>>>>>
>>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>>>>>>> RUNNING. The
>>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>
>>>>>>> I see. If this is the intention, than I would suggest to have two
>>> (or
>>>>>>> maybe three) different exceptions:
>>>>>>>
>>>>>>> - StreamThreadRebalancingException (retryable)
>>>>>>> - StreamThreadNotRunning (not retryable -- thrown if in state
>>>>>>> PENDING_SHUTDOWN or DEAD
>>>>>>> - maybe StreamThreadNotStarted (for state CREATED)
>>>>>>>
>>>>>>> The last one is tricky and could also be merged into one of the
>>> first
>>>>>>> two, depending if you want to argue that it's retryable or not.
>>> (Just
>>>>>>> food for though -- not sure what others think.)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 6/22/18 8:06 AM, vito jeng wrote:
>>>>>>>> Matthias,
>>>>>>>>
>>>>>>>> Thank you for your assistance.
>>>>>>>>
>>>>>>>>> what is the status of this KIP?
>>>>>>>>
>>>>>>>> Unfortunately, there is no further progress.
>>>>>>>> About seven weeks ago, I was injured in sports. I had a broken
>>> wrist
>>>> on
>>>>>>>> my left wrist.
>>>>>>>> Many jobs are affected, including this KIP and implementation.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>>> do
>>>> we
>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>> really
>>>>> need
>>>>>>>>> them? Can't we just throw the correct exception directly instead
>>> of
>>>>>>>>> wrapping it later?
>>>>>>>>
>>>>>>>> I think you may be right. As I say in the previous:
>>>>>>>> "The original idea is that we can distinguish different state
>>> store
>>>>>>>> exception for different handling. But to be honest, I am not quite
>>>> sure
>>>>>>>> this is necessary. Maybe have some change during implementation."
>>>>>>>>
>>>>>>>> During the implementation, I also feel we maybe not need wrapper
>>> it.
>>>>>>>> We can just throw the correctly directly.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>
>>>>>>>> When KafkaStream state is REBALANCING, I think it is a retryable
>>>> error.
>>>>>>>>
>>>>>>>> StreamThreadStateStoreProvider#stores() will throw
>>>>>>>> StreamThreadNotRunningException when StreamThread state is not
>>>>> RUNNING.
>>>>>>> The
>>>>>>>> user can retry until KafkaStream state is RUNNING.
>>>>>>>>
>>>>>>>>
>>>>>>>>> When would we throw an `StateStoreEmptyException`? The semantics
>>> is
>>>>>>>> unclear to me atm.
>>>>>>>>
>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>> retryable
>>>>>>>> error?
>>>>>>>>
>>>>>>>> These two comments will be answered in another mail.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>>>>> matthias@confluent.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Vito,
>>>>>>>>>
>>>>>>>>> what is the status of this KIP?
>>>>>>>>>
>>>>>>>>> I just re-read it, and have a couple of follow up comments. Why
>>> do
>>>> we
>>>>>>>>> discuss the internal exceptions you want to add? Also, do we
>>> really
>>>>> need
>>>>>>>>> them? Can't we just throw the correct exception directly instead
>>> of
>>>>>>>>> wrapping it later?
>>>>>>>>>
>>>>>>>>> When would we throw an `StateStoreEmptyException`? The semantics
>>> is
>>>>>>>>> unclear to me atm.
>>>>>>>>>
>>>>>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>>>>>>
>>>>>>>>> When the state is RUNNING, is `StateStoreClosedException` a
>>>> retryable
>>>>>>>>> error?
>>>>>>>>>
>>>>>>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key,
>>>> long
>>>>>>>>> time); that should be added
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Overall I like the KIP but some details are still unclear. Maybe
>>> it
>>>>>>>>> might help if you open an PR in parallel?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>>>>>>>>>> Hi, Guozhang,
>>>>>>>>>>
>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi, Bill,
>>>>>>>>>>
>>>>>>>>>> I'll try to make some update to make the KIP better.
>>>>>>>>>>
>>>>>>>>>> Thanks for the comment!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bbejeck@gmail.com
>>>>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the KIP, overall it's a +1 from me.
>>>>>>>>>>>
>>>>>>>>>>> At this point, the only thing I would change is possibly
>>> removing
>>>>> the
>>>>>>>>>>> listing of all methods called by the user and the listing of
>>> all
>>>>> store
>>>>>>>>>>> types and focus on what states result in which exceptions
>>> thrown
>>>> to
>>>>>>> the
>>>>>>>>>>> user.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Bill
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>>>> wangguoz@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>
>>>>>>>>>>>> I made a pass over the wiki and it looks great to me. I'm +1
>>> on
>>>> the
>>>>>>>>> KIP.
>>>>>>>>>>>>
>>>>>>>>>>>> About the base class InvalidStateStoreException itself, I'd
>>>>> actually
>>>>>>>>>>>> suggest we do not deprecate it but still expose it as part of
>>> the
>>>>>>>>> public
>>>>>>>>>>>> API, for people who do not want to handle these cases
>>> differently
>>>>> (if
>>>>>>>>> we
>>>>>>>>>>>> deprecate it then we are enforcing them to capture all three
>>>>>>> exceptions
>>>>>>>>>>>> one-by-one).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>>> john@confluent.io
>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Vito,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think it's much nicer to give callers different exceptions
>>> to
>>>>> tell
>>>>>>>>>>> them
>>>>>>>>>>>>> whether the state store got migrated, whether it's still
>>>>>>> initializing,
>>>>>>>>>>> or
>>>>>>>>>>>>> whether there's some unrecoverable error.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In the KIP, it's typically not necessary to discuss
>>>>> non-user-facing
>>>>>>>>>>>> details
>>>>>>>>>>>>> such as what exceptions we will throw internally. The KIP is
>>>>>>> primarily
>>>>>>>>>>> to
>>>>>>>>>>>>> discuss public interface changes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You might consider simply removing all the internal details
>>> from
>>>>> the
>>>>>>>>>>> KIP,
>>>>>>>>>>>>> which will have the dual advantage that it makes the KIP
>>> smaller
>>>>> and
>>>>>>>>>>>> easier
>>>>>>>>>>>>> to agree on, as well as giving you more freedom in the
>>> internal
>>>>>>>>> details
>>>>>>>>>>>>> when it comes to implementation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I like your decision to have your refined exceptions extend
>>>>>>>>>>>>> InvalidStateStoreException to ensure backward compatibility.
>>>> Since
>>>>>>> we
>>>>>>>>>>>> want
>>>>>>>>>>>>> to encourage callers to catch the more specific exceptions,
>>> and
>>>> we
>>>>>>>>>>> don't
>>>>>>>>>>>>> expect to ever throw a raw InvalidStateStoreException
>>> anymore,
>>>> you
>>>>>>>>>>> might
>>>>>>>>>>>>> consider adding the @Deprecated annotation to
>>>>>>>>>>> InvalidStateStoreException.
>>>>>>>>>>>>> This will gently encourage callers to migrate to the new
>>>> exception
>>>>>>> and
>>>>>>>>>>>> open
>>>>>>>>>>>>> the possibility of removing InvalidStateStoreException
>>> entirely
>>>>> in a
>>>>>>>>>>>> future
>>>>>>>>>>>>> release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> -John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for clarification! That makes sense to me.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you update the KIP to make those suggestions explicit?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Got it!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
>>>> you
>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>>>> anymore,
>>>>>>>>>>> but
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
>>> thrown
>>>>> at
>>>>>>>>>>> all
>>>>>>>>>>>>>>> anymore,
>>>>>>>>>>>>>>> but only new sub-classes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>> exception
>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will
>>>>> throw
>>>>>>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>>>>>>>>>>> And then the internal exception will be wrapped as
>>>>>>>>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException`
>>>>> during
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
>>>> It's
>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>>>>> distinguish
>>>>>>>>>>>>> between
>>>>>>>>>>>>>>> the different kinds of InvalidStateStoreException.
>>>>>>>>>>>>>>> The original idea is that we can distinguish different
>>> state
>>>>> store
>>>>>>>>>>>>>>> exception for
>>>>>>>>>>>>>>> different handling.
>>>>>>>>>>>>>>> But to be honest, I am not quite sure this is necessary.
>>> Maybe
>>>>>>> have
>>>>>>>>>>>>>>> some change during implementation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Does it make sense?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the update Vito!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
>>>> you
>>>>>>>>>>>> suggest
>>>>>>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>>>> anymore,
>>>>>>>>>>> but
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The internal exception will be wrapped as category
>>> exception
>>>>>>>>>>>> finally.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
>>>> It's
>>>>>>>>>>>> unclear
>>>>>>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the review.
>>>>>>>>>>>>>>>>> I reply separately in the following sections.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>>>>>>>>>>> matthias@confluent.io
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>>> pause...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Seems you did a very thorough investigation of the code.
>>>> It's
>>>>>>>>>>>> useful
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> understand what user facing interfaces are affected.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I also think too detailed. Especially the section
>>> `Changes
>>>> in
>>>>>>>>>>> call
>>>>>>>>>>>>>>>> trace`.
>>>>>>>>>>>>>>>>> Do you think it should be removed?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> To summarize my current understanding of your KIP, the
>>> main
>>>>>>>>>>> change
>>>>>>>>>>>>> is
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> introduce new exceptions that extend
>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
>>>>>>>>>>>>>>>>> I think the best way is that all new exceptions extend
>>> from
>>>>>>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Some questions:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Why do we need ```? Could
>>> `InvalidStateStoreException`
>>>> be
>>>>>>>>>>> used
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> this purpose?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Does this question miss some word?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - What the superclass of
>>> `StateStoreStreamThreadNotRunni
>>>>>>>>>>>>> ngException`
>>>>>>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception
>>> ?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I apologize for not well written parts. I tried to modify
>>>> some
>>>>>>>>>>> code
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> recent period and modify the KIP.
>>>>>>>>>>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think you are right. We need to look at the state
>>> both
>>>>> of
>>>>>>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread
>>> and
>>>>> state
>>>>>>>>>>>>> store,
>>>>>>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not
>>>>> hosted
>>>>>>>>>>> an
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>>> state
>>>>>>>>>>>> rebalance
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
>>> store.
>>>>> For
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> second
>>>>>>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>>> PENDING_SHUTDOWN, or
>>>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>>>> rediscover
>>>>> or
>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
>>> user
>>>> can
>>>>>>>>>>> just
>>>>>>>>>>>>>> plain
>>>>>>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
>>>>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>>>>>>> state store migrated to another instance, the user
>>>> needs
>>>>>>> to
>>>>>>>>>>>>>>>>>> rediscover
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>>>>>>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>>> idea of
>>>>> the
>>>>>>>>>>>> Jira
>>>>>>>>>>>>>>>> was,
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
>>>>>>>>>>>> strategies
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a
>>>> user
>>>>>>>>>>> just
>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand,
>>> if
>>>> a
>>>>>>>>>>> store
>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover
>>> the
>>>>> store
>>>>>>>>>>>>>> instead
>>>>>>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>>> talks
>>>>> about
>>>>>>>>>>>> this
>>>>>>>>>>>>>>>> ideas
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure
>>>> out
>>>>>>>>>>> first
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>>> users, if
>>>>> we
>>>>>>>>>>>> only
>>>>>>>>>>>>>>>> wants
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we
>>>>> should
>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>>>>>>>>>>> StateStoreClosedException
>>>>>>>>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
>>>>> possible
>>>>>>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call
>>>>> trace,
>>>>>>>>>>> and
>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and
>>>> which
>>>>>>>>>>> ones
>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>>> should
>>>> not
>>>>>>> be
>>>>>>>>>>>>>> wrapped
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>>>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>>>>> confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>>>>>>>>>>> different+errors
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> -- Guozhang
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -- Guozhang
>>>>
>>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Just open a PR for further discussion:
https://github.com/apache/kafka/pull/5814
Any suggestion is welcome.
Thanks!
---
Vito
On Thu, Oct 11, 2018 at 12:14 AM vito jeng <vi...@is-land.com.tw> wrote:
> Hi John,
>
> Thanks for reviewing the KIP.
>
> > I didn't follow the addition of a new method to the QueryableStoreType
> > interface. Can you elaborate why this is necessary to support the new
> > exception types?
>
> To support the new exception types, I would check stream state in the
> following classes:
> - CompositeReadOnlyKeyValueStore class
> - CompositeReadOnlySessionStore class
> - CompositeReadOnlyWindowStore class
> - DelegatingPeekingKeyValueIterator class
>
> It is also necessary to keep backward compatibility. So I plan passing
> stream
> instance to QueryableStoreType instance during KafkaStreams#store()
> invoked.
> It looks a most simple way, I think.
>
> It is why I add a new method to the QueryableStoreType interface. I can
> understand
> that we should try to avoid adding the public api method. However, at the
> moment
> I have no better ideas.
>
> Any thoughts?
>
>
> > Also, looking over your KIP again, it seems valuable to introduce
> > "retriable store exception" and "fatal store exception" marker interfaces
> > that the various exceptions can mix in. It would be nice from a usability
> > perspective to be able to just log and retry on any "retriable" exception
> > and log and shutdown on any fatal exception.
>
> I agree that this is valuable to the user.
> I'll update the KIP.
>
>
> Thanks
>
>
> ---
> Vito
>
>
> On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io> wrote:
>
>> Hi Vito,
>>
>> I'm glad to hear you're well again!
>>
>> I didn't follow the addition of a new method to the QueryableStoreType
>> interface. Can you elaborate why this is necessary to support the new
>> exception types?
>>
>> Also, looking over your KIP again, it seems valuable to introduce
>> "retriable store exception" and "fatal store exception" marker interfaces
>> that the various exceptions can mix in. It would be nice from a usability
>> perspective to be able to just log and retry on any "retriable" exception
>> and log and shutdown on any fatal exception.
>>
>> Thanks,
>> -John
>>
>> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com> wrote:
>>
>> > Thanks for the explanation, that makes sense.
>> >
>> >
>> > Guozhang
>> >
>> >
>> > On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <matthias@confluent.io
>> >
>> > wrote:
>> >
>> > > The scenario I had I mind was, that KS is started in one thread while
>> a
>> > > second thread has a reference to the object to issue queries.
>> > >
>> > > If a query is issue before the "main thread" started KS, and the
>> "query
>> > > thread" knows that it will eventually get started, it can retry. On
>> the
>> > > other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
>> impossible
>> > > to issue any query against it now or in the future and thus the error
>> is
>> > > not retryable.
>> > >
>> > >
>> > > -Matthias
>> > >
>> > > On 6/25/18 10:15 AM, Guozhang Wang wrote:
>> > > > I'm wondering if StreamThreadNotStarted could be merged into
>> > > > StreamThreadNotRunning, because I think users' handling logic for
>> the
>> > > third
>> > > > case would be likely the same as the second. Do you have some
>> scenarios
>> > > > where users may want to handle them differently?
>> > > >
>> > > > Guozhang
>> > > >
>> > > > On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
>> > matthias@confluent.io>
>> > > > wrote:
>> > > >
>> > > >> Sorry to hear! Get well soon!
>> > > >>
>> > > >> It's not a big deal if the KIP stalls a little bit. Feel free to
>> pick
>> > it
>> > > >> up again when you find time.
>> > > >>
>> > > >>>>> Is `StreamThreadNotRunningException` really an retryable error?
>> > > >>>>
>> > > >>>> When KafkaStream state is REBALANCING, I think it is a retryable
>> > > error.
>> > > >>>>
>> > > >>>> StreamThreadStateStoreProvider#stores() will throw
>> > > >>>> StreamThreadNotRunningException when StreamThread state is not
>> > > >> RUNNING. The
>> > > >>>> user can retry until KafkaStream state is RUNNING.
>> > > >>
>> > > >> I see. If this is the intention, than I would suggest to have two
>> (or
>> > > >> maybe three) different exceptions:
>> > > >>
>> > > >> - StreamThreadRebalancingException (retryable)
>> > > >> - StreamThreadNotRunning (not retryable -- thrown if in state
>> > > >> PENDING_SHUTDOWN or DEAD
>> > > >> - maybe StreamThreadNotStarted (for state CREATED)
>> > > >>
>> > > >> The last one is tricky and could also be merged into one of the
>> first
>> > > >> two, depending if you want to argue that it's retryable or not.
>> (Just
>> > > >> food for though -- not sure what others think.)
>> > > >>
>> > > >>
>> > > >>
>> > > >> -Matthias
>> > > >>
>> > > >> On 6/22/18 8:06 AM, vito jeng wrote:
>> > > >>> Matthias,
>> > > >>>
>> > > >>> Thank you for your assistance.
>> > > >>>
>> > > >>>> what is the status of this KIP?
>> > > >>>
>> > > >>> Unfortunately, there is no further progress.
>> > > >>> About seven weeks ago, I was injured in sports. I had a broken
>> wrist
>> > on
>> > > >>> my left wrist.
>> > > >>> Many jobs are affected, including this KIP and implementation.
>> > > >>>
>> > > >>>
>> > > >>>> I just re-read it, and have a couple of follow up comments. Why
>> do
>> > we
>> > > >>>> discuss the internal exceptions you want to add? Also, do we
>> really
>> > > need
>> > > >>>> them? Can't we just throw the correct exception directly instead
>> of
>> > > >>>> wrapping it later?
>> > > >>>
>> > > >>> I think you may be right. As I say in the previous:
>> > > >>> "The original idea is that we can distinguish different state
>> store
>> > > >>> exception for different handling. But to be honest, I am not quite
>> > sure
>> > > >>> this is necessary. Maybe have some change during implementation."
>> > > >>>
>> > > >>> During the implementation, I also feel we maybe not need wrapper
>> it.
>> > > >>> We can just throw the correctly directly.
>> > > >>>
>> > > >>>
>> > > >>>> Is `StreamThreadNotRunningException` really an retryable error?
>> > > >>>
>> > > >>> When KafkaStream state is REBALANCING, I think it is a retryable
>> > error.
>> > > >>>
>> > > >>> StreamThreadStateStoreProvider#stores() will throw
>> > > >>> StreamThreadNotRunningException when StreamThread state is not
>> > > RUNNING.
>> > > >> The
>> > > >>> user can retry until KafkaStream state is RUNNING.
>> > > >>>
>> > > >>>
>> > > >>>> When would we throw an `StateStoreEmptyException`? The semantics
>> is
>> > > >>> unclear to me atm.
>> > > >>>
>> > > >>>> When the state is RUNNING, is `StateStoreClosedException` a
>> > retryable
>> > > >>> error?
>> > > >>>
>> > > >>> These two comments will be answered in another mail.
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>> ---
>> > > >>> Vito
>> > > >>>
>> > > >>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
>> > > matthias@confluent.io>
>> > > >>> wrote:
>> > > >>>
>> > > >>>> Vito,
>> > > >>>>
>> > > >>>> what is the status of this KIP?
>> > > >>>>
>> > > >>>> I just re-read it, and have a couple of follow up comments. Why
>> do
>> > we
>> > > >>>> discuss the internal exceptions you want to add? Also, do we
>> really
>> > > need
>> > > >>>> them? Can't we just throw the correct exception directly instead
>> of
>> > > >>>> wrapping it later?
>> > > >>>>
>> > > >>>> When would we throw an `StateStoreEmptyException`? The semantics
>> is
>> > > >>>> unclear to me atm.
>> > > >>>>
>> > > >>>> Is `StreamThreadNotRunningException` really an retryable error?
>> > > >>>>
>> > > >>>> When the state is RUNNING, is `StateStoreClosedException` a
>> > retryable
>> > > >>>> error?
>> > > >>>>
>> > > >>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key,
>> > long
>> > > >>>> time); that should be added
>> > > >>>>
>> > > >>>>
>> > > >>>> Overall I like the KIP but some details are still unclear. Maybe
>> it
>> > > >>>> might help if you open an PR in parallel?
>> > > >>>>
>> > > >>>>
>> > > >>>> -Matthias
>> > > >>>>
>> > > >>>> On 4/24/18 8:18 AM, vito jeng wrote:
>> > > >>>>> Hi, Guozhang,
>> > > >>>>>
>> > > >>>>> Thanks for the comment!
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> Hi, Bill,
>> > > >>>>>
>> > > >>>>> I'll try to make some update to make the KIP better.
>> > > >>>>>
>> > > >>>>> Thanks for the comment!
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> ---
>> > > >>>>> Vito
>> > > >>>>>
>> > > >>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bbejeck@gmail.com
>> >
>> > > >> wrote:
>> > > >>>>>
>> > > >>>>>> Hi Vito,
>> > > >>>>>>
>> > > >>>>>> Thanks for the KIP, overall it's a +1 from me.
>> > > >>>>>>
>> > > >>>>>> At this point, the only thing I would change is possibly
>> removing
>> > > the
>> > > >>>>>> listing of all methods called by the user and the listing of
>> all
>> > > store
>> > > >>>>>> types and focus on what states result in which exceptions
>> thrown
>> > to
>> > > >> the
>> > > >>>>>> user.
>> > > >>>>>>
>> > > >>>>>> Thanks,
>> > > >>>>>> Bill
>> > > >>>>>>
>> > > >>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
>> > wangguoz@gmail.com>
>> > > >>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> Thanks for the KIP Vito!
>> > > >>>>>>>
>> > > >>>>>>> I made a pass over the wiki and it looks great to me. I'm +1
>> on
>> > the
>> > > >>>> KIP.
>> > > >>>>>>>
>> > > >>>>>>> About the base class InvalidStateStoreException itself, I'd
>> > > actually
>> > > >>>>>>> suggest we do not deprecate it but still expose it as part of
>> the
>> > > >>>> public
>> > > >>>>>>> API, for people who do not want to handle these cases
>> differently
>> > > (if
>> > > >>>> we
>> > > >>>>>>> deprecate it then we are enforcing them to capture all three
>> > > >> exceptions
>> > > >>>>>>> one-by-one).
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> Guozhang
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
>> john@confluent.io
>> > >
>> > > >>>> wrote:
>> > > >>>>>>>
>> > > >>>>>>>> Hi Vito,
>> > > >>>>>>>>
>> > > >>>>>>>> Thanks for the KIP!
>> > > >>>>>>>>
>> > > >>>>>>>> I think it's much nicer to give callers different exceptions
>> to
>> > > tell
>> > > >>>>>> them
>> > > >>>>>>>> whether the state store got migrated, whether it's still
>> > > >> initializing,
>> > > >>>>>> or
>> > > >>>>>>>> whether there's some unrecoverable error.
>> > > >>>>>>>>
>> > > >>>>>>>> In the KIP, it's typically not necessary to discuss
>> > > non-user-facing
>> > > >>>>>>> details
>> > > >>>>>>>> such as what exceptions we will throw internally. The KIP is
>> > > >> primarily
>> > > >>>>>> to
>> > > >>>>>>>> discuss public interface changes.
>> > > >>>>>>>>
>> > > >>>>>>>> You might consider simply removing all the internal details
>> from
>> > > the
>> > > >>>>>> KIP,
>> > > >>>>>>>> which will have the dual advantage that it makes the KIP
>> smaller
>> > > and
>> > > >>>>>>> easier
>> > > >>>>>>>> to agree on, as well as giving you more freedom in the
>> internal
>> > > >>>> details
>> > > >>>>>>>> when it comes to implementation.
>> > > >>>>>>>>
>> > > >>>>>>>> I like your decision to have your refined exceptions extend
>> > > >>>>>>>> InvalidStateStoreException to ensure backward compatibility.
>> > Since
>> > > >> we
>> > > >>>>>>> want
>> > > >>>>>>>> to encourage callers to catch the more specific exceptions,
>> and
>> > we
>> > > >>>>>> don't
>> > > >>>>>>>> expect to ever throw a raw InvalidStateStoreException
>> anymore,
>> > you
>> > > >>>>>> might
>> > > >>>>>>>> consider adding the @Deprecated annotation to
>> > > >>>>>> InvalidStateStoreException.
>> > > >>>>>>>> This will gently encourage callers to migrate to the new
>> > exception
>> > > >> and
>> > > >>>>>>> open
>> > > >>>>>>>> the possibility of removing InvalidStateStoreException
>> entirely
>> > > in a
>> > > >>>>>>> future
>> > > >>>>>>>> release.
>> > > >>>>>>>>
>> > > >>>>>>>> Thanks,
>> > > >>>>>>>> -John
>> > > >>>>>>>>
>> > > >>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>> > > >>>>>> matthias@confluent.io>
>> > > >>>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>>> Thanks for clarification! That makes sense to me.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Can you update the KIP to make those suggestions explicit?
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> -Matthias
>> > > >>>>>>>>>
>> > > >>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>> > > >>>>>>>>>> Matthias,
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Thanks for the feedback!
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Got it!
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> The (incomplete) question was, if we need
>> > > >>>>>> `StateStoreFailException`
>> > > >>>>>>> or
>> > > >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
>> > you
>> > > >>>>>>> suggest
>> > > >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>> > anymore,
>> > > >>>>>> but
>> > > >>>>>>>> only
>> > > >>>>>>>>>>> the new sub-classes (just to get a better understanding).
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
>> thrown
>> > > at
>> > > >>>>>> all
>> > > >>>>>>>>>> anymore,
>> > > >>>>>>>>>> but only new sub-classes.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Not sure what this sentence means:
>> > > >>>>>>>>>>>> The internal exception will be wrapped as category
>> exception
>> > > >>>>>>> finally.
>> > > >>>>>>>>>>> Can you elaborate?
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will
>> > > throw
>> > > >>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>> > > >>>>>>>>>> And then the internal exception will be wrapped as
>> > > >>>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException`
>> > > during
>> > > >>>>>>> the
>> > > >>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
>> > It's
>> > > >>>>>>> unclear
>> > > >>>>>>>>>> to me atm why they are introduced.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
>> > > distinguish
>> > > >>>>>>>> between
>> > > >>>>>>>>>> the different kinds of InvalidStateStoreException.
>> > > >>>>>>>>>> The original idea is that we can distinguish different
>> state
>> > > store
>> > > >>>>>>>>>> exception for
>> > > >>>>>>>>>> different handling.
>> > > >>>>>>>>>> But to be honest, I am not quite sure this is necessary.
>> Maybe
>> > > >> have
>> > > >>>>>>>>>> some change during implementation.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Does it make sense?
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> ---
>> > > >>>>>>>>>> Vito
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>> > > >>>>>>>> matthias@confluent.io>
>> > > >>>>>>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> Thanks for the update Vito!
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> The (incomplete) question was, if we need
>> > > >>>>>> `StateStoreFailException`
>> > > >>>>>>> or
>> > > >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
>> > you
>> > > >>>>>>> suggest
>> > > >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
>> > anymore,
>> > > >>>>>> but
>> > > >>>>>>>> only
>> > > >>>>>>>>>>> the new sub-classes (just to get a better understanding).
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Not sure what this sentence means:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> The internal exception will be wrapped as category
>> exception
>> > > >>>>>>> finally.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Can you elaborate?
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
>> > It's
>> > > >>>>>>> unclear
>> > > >>>>>>>>>>> to me atm why they are introduced.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> -Matthias
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>> > > >>>>>>>>>>>> Matthias,
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Thanks for the review.
>> > > >>>>>>>>>>>> I reply separately in the following sections.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> ---
>> > > >>>>>>>>>>>> Vito
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>> > > >>>>>>>> matthias@confluent.io
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
>> pause...
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Seems you did a very thorough investigation of the code.
>> > It's
>> > > >>>>>>> useful
>> > > >>>>>>>>> to
>> > > >>>>>>>>>>>>> understand what user facing interfaces are affected.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> I also think too detailed. Especially the section
>> `Changes
>> > in
>> > > >>>>>> call
>> > > >>>>>>>>>>> trace`.
>> > > >>>>>>>>>>>> Do you think it should be removed?
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> To summarize my current understanding of your KIP, the
>> main
>> > > >>>>>> change
>> > > >>>>>>>> is
>> > > >>>>>>>>> to
>> > > >>>>>>>>>>>>> introduce new exceptions that extend
>> > > >>>>>> `InvalidStateStoreException`.
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
>> > > >>>>>>>>>>>> I think the best way is that all new exceptions extend
>> from
>> > > >>>>>>>>>>>> `InvalidStateStoreException`.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Some questions:
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> - Why do we need ```? Could
>> `InvalidStateStoreException`
>> > be
>> > > >>>>>> used
>> > > >>>>>>>> for
>> > > >>>>>>>>>>>>> this purpose?
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Does this question miss some word?
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> - What the superclass of
>> `StateStoreStreamThreadNotRunni
>> > > >>>>>>>> ngException`
>> > > >>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>> > > >>>>>>>>>>> `StateStoreFailException`
>> > > >>>>>>>>>>>>> ?
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception
>> ?
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>> I apologize for not well written parts. I tried to modify
>> > some
>> > > >>>>>> code
>> > > >>>>>>>> in
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>> recent period and modify the KIP.
>> > > >>>>>>>>>>>> The modification is now complete. Please look again.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> -Matthias
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>> > > >>>>>>>>>>>>>> Matthias,
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Sorry for not response these days.
>> > > >>>>>>>>>>>>>> I just finished it. Please have a look. :)
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> ---
>> > > >>>>>>>>>>>>>> Vito
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>> > > >>>>>>>>>>> matthias@confluent.io>
>> > > >>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Is there any update on this KIP?
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> -Matthias
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>> > > >>>>>>>>>>>>>>>> Matthias,
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Thank you for your response.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> I think you are right. We need to look at the state
>> both
>> > > of
>> > > >>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread
>> and
>> > > state
>> > > >>>>>>>> store,
>> > > >>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> ---
>> > > >>>>>>>>>>>>>>>> Vito
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>> > > >>>>>>>>>>>>> matthias@confluent.io>
>> > > >>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Vito,
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Sorry for this late reply.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> There can be two cases:
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not
>> > > hosted
>> > > >>>>>> an
>> > > >>>>>>>> the
>> > > >>>>>>>>>>>>>>>>> application instance anymore,
>> > > >>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in
>> state
>> > > >>>>>>> rebalance
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
>> store.
>> > > For
>> > > >>>>>>> the
>> > > >>>>>>>>>>> second
>> > > >>>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR,
>> PENDING_SHUTDOWN, or
>> > > >>>>>>>>> NOT_RUNNING,
>> > > >>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
>> > rediscover
>> > > or
>> > > >>>>>>>> retry.
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> Does this make sense?
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> -Matthias
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>> > > >>>>>>>>>>>>>>>>>> Matthias,
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> I try to clarify some concept.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the
>> user
>> > can
>> > > >>>>>> just
>> > > >>>>>>>>> plain
>> > > >>>>>>>>>>>>>>>>> retry.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
>> > > >>>>>>> NOT_RUNNING,
>> > > >>>>>>>>> it
>> > > >>>>>>>>>>>>>>> means
>> > > >>>>>>>>>>>>>>>>>> state store migrated to another instance, the user
>> > needs
>> > > >> to
>> > > >>>>>>>>>>>>> rediscover
>> > > >>>>>>>>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>> store.
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> Is my understanding correct?
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> ---
>> > > >>>>>>>>>>>>>>>>>> Vito
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>> > > >>>>>>>>>>>>>>> matthias@confluent.io>
>> > > >>>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original
>> idea of
>> > > the
>> > > >>>>>>> Jira
>> > > >>>>>>>>>>> was,
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
>> > > >>>>>>> strategies
>> > > >>>>>>>> to
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>> user.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a
>> > user
>> > > >>>>>> just
>> > > >>>>>>>> need
>> > > >>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>> wait
>> > > >>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand,
>> if
>> > a
>> > > >>>>>> store
>> > > >>>>>>>> go
>> > > >>>>>>>>>>>>>>> migrated
>> > > >>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover
>> the
>> > > store
>> > > >>>>>>>>> instead
>> > > >>>>>>>>>>>>> of a
>> > > >>>>>>>>>>>>>>>>>>> "plain retry".
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that
>> talks
>> > > about
>> > > >>>>>>> this
>> > > >>>>>>>>>>> ideas
>> > > >>>>>>>>>>>>>>> and
>> > > >>>>>>>>>>>>>>>>>>> reasoning.
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> -Matthias
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>> > > >>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure
>> > out
>> > > >>>>>> first
>> > > >>>>>>>> is
>> > > >>>>>>>>>>> what
>> > > >>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the
>> users, if
>> > > we
>> > > >>>>>>> only
>> > > >>>>>>>>>>> wants
>> > > >>>>>>>>>>>>> to
>> > > >>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we
>> > > should
>> > > >>>>>>>> rename
>> > > >>>>>>>>>>> the
>> > > >>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>> > > >>>>>>>>> StateStoreClosedException
>> > > >>>>>>>>>>>>>>>>> classes.
>> > > >>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
>> > > possible
>> > > >>>>>>>>> internal
>> > > >>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call
>> > > trace,
>> > > >>>>>> and
>> > > >>>>>>>>> which
>> > > >>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and
>> > which
>> > > >>>>>> ones
>> > > >>>>>>>>>>> should
>> > > >>>>>>>>>>>>> be
>> > > >>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones
>> should
>> > not
>> > > >> be
>> > > >>>>>>>>> wrapped
>> > > >>>>>>>>>>>>> at
>> > > >>>>>>>>>>>>>>>>> all
>> > > >>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> Guozhang
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>> > > >>>>>>>>> vito@is-land.com.tw>
>> > > >>>>>>>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Hi,
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
>> > > confluence/display/KAFKA/KIP-
>> > > >>>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>> > > >>>>>>>>>>> different+errors
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> Please have a look.
>> > > >>>>>>>>>>>>>>>>>>>>> Thanks!
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>> ---
>> > > >>>>>>>>>>>>>>>>>>>>> Vito
>> > > >>>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>>
>> > > >>>>>>> --
>> > > >>>>>>> -- Guozhang
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>> > --
>> > -- Guozhang
>> >
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Hi John,
Thanks for reviewing the KIP.
> I didn't follow the addition of a new method to the QueryableStoreType
> interface. Can you elaborate why this is necessary to support the new
> exception types?
To support the new exception types, I would check stream state in the
following classes:
- CompositeReadOnlyKeyValueStore class
- CompositeReadOnlySessionStore class
- CompositeReadOnlyWindowStore class
- DelegatingPeekingKeyValueIterator class
It is also necessary to keep backward compatibility. So I plan passing
stream
instance to QueryableStoreType instance during KafkaStreams#store()
invoked.
It looks a most simple way, I think.
It is why I add a new method to the QueryableStoreType interface. I can
understand
that we should try to avoid adding the public api method. However, at the
moment
I have no better ideas.
Any thoughts?
> Also, looking over your KIP again, it seems valuable to introduce
> "retriable store exception" and "fatal store exception" marker interfaces
> that the various exceptions can mix in. It would be nice from a usability
> perspective to be able to just log and retry on any "retriable" exception
> and log and shutdown on any fatal exception.
I agree that this is valuable to the user.
I'll update the KIP.
Thanks
---
Vito
On Tue, Oct 9, 2018 at 2:30 AM John Roesler <jo...@confluent.io> wrote:
> Hi Vito,
>
> I'm glad to hear you're well again!
>
> I didn't follow the addition of a new method to the QueryableStoreType
> interface. Can you elaborate why this is necessary to support the new
> exception types?
>
> Also, looking over your KIP again, it seems valuable to introduce
> "retriable store exception" and "fatal store exception" marker interfaces
> that the various exceptions can mix in. It would be nice from a usability
> perspective to be able to just log and retry on any "retriable" exception
> and log and shutdown on any fatal exception.
>
> Thanks,
> -John
>
> On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com> wrote:
>
> > Thanks for the explanation, that makes sense.
> >
> >
> > Guozhang
> >
> >
> > On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> > > The scenario I had I mind was, that KS is started in one thread while a
> > > second thread has a reference to the object to issue queries.
> > >
> > > If a query is issue before the "main thread" started KS, and the "query
> > > thread" knows that it will eventually get started, it can retry. On the
> > > other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is
> impossible
> > > to issue any query against it now or in the future and thus the error
> is
> > > not retryable.
> > >
> > >
> > > -Matthias
> > >
> > > On 6/25/18 10:15 AM, Guozhang Wang wrote:
> > > > I'm wondering if StreamThreadNotStarted could be merged into
> > > > StreamThreadNotRunning, because I think users' handling logic for the
> > > third
> > > > case would be likely the same as the second. Do you have some
> scenarios
> > > > where users may want to handle them differently?
> > > >
> > > > Guozhang
> > > >
> > > > On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
> > matthias@confluent.io>
> > > > wrote:
> > > >
> > > >> Sorry to hear! Get well soon!
> > > >>
> > > >> It's not a big deal if the KIP stalls a little bit. Feel free to
> pick
> > it
> > > >> up again when you find time.
> > > >>
> > > >>>>> Is `StreamThreadNotRunningException` really an retryable error?
> > > >>>>
> > > >>>> When KafkaStream state is REBALANCING, I think it is a retryable
> > > error.
> > > >>>>
> > > >>>> StreamThreadStateStoreProvider#stores() will throw
> > > >>>> StreamThreadNotRunningException when StreamThread state is not
> > > >> RUNNING. The
> > > >>>> user can retry until KafkaStream state is RUNNING.
> > > >>
> > > >> I see. If this is the intention, than I would suggest to have two
> (or
> > > >> maybe three) different exceptions:
> > > >>
> > > >> - StreamThreadRebalancingException (retryable)
> > > >> - StreamThreadNotRunning (not retryable -- thrown if in state
> > > >> PENDING_SHUTDOWN or DEAD
> > > >> - maybe StreamThreadNotStarted (for state CREATED)
> > > >>
> > > >> The last one is tricky and could also be merged into one of the
> first
> > > >> two, depending if you want to argue that it's retryable or not.
> (Just
> > > >> food for though -- not sure what others think.)
> > > >>
> > > >>
> > > >>
> > > >> -Matthias
> > > >>
> > > >> On 6/22/18 8:06 AM, vito jeng wrote:
> > > >>> Matthias,
> > > >>>
> > > >>> Thank you for your assistance.
> > > >>>
> > > >>>> what is the status of this KIP?
> > > >>>
> > > >>> Unfortunately, there is no further progress.
> > > >>> About seven weeks ago, I was injured in sports. I had a broken
> wrist
> > on
> > > >>> my left wrist.
> > > >>> Many jobs are affected, including this KIP and implementation.
> > > >>>
> > > >>>
> > > >>>> I just re-read it, and have a couple of follow up comments. Why do
> > we
> > > >>>> discuss the internal exceptions you want to add? Also, do we
> really
> > > need
> > > >>>> them? Can't we just throw the correct exception directly instead
> of
> > > >>>> wrapping it later?
> > > >>>
> > > >>> I think you may be right. As I say in the previous:
> > > >>> "The original idea is that we can distinguish different state store
> > > >>> exception for different handling. But to be honest, I am not quite
> > sure
> > > >>> this is necessary. Maybe have some change during implementation."
> > > >>>
> > > >>> During the implementation, I also feel we maybe not need wrapper
> it.
> > > >>> We can just throw the correctly directly.
> > > >>>
> > > >>>
> > > >>>> Is `StreamThreadNotRunningException` really an retryable error?
> > > >>>
> > > >>> When KafkaStream state is REBALANCING, I think it is a retryable
> > error.
> > > >>>
> > > >>> StreamThreadStateStoreProvider#stores() will throw
> > > >>> StreamThreadNotRunningException when StreamThread state is not
> > > RUNNING.
> > > >> The
> > > >>> user can retry until KafkaStream state is RUNNING.
> > > >>>
> > > >>>
> > > >>>> When would we throw an `StateStoreEmptyException`? The semantics
> is
> > > >>> unclear to me atm.
> > > >>>
> > > >>>> When the state is RUNNING, is `StateStoreClosedException` a
> > retryable
> > > >>> error?
> > > >>>
> > > >>> These two comments will be answered in another mail.
> > > >>>
> > > >>>
> > > >>>
> > > >>> ---
> > > >>> Vito
> > > >>>
> > > >>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
> > > matthias@confluent.io>
> > > >>> wrote:
> > > >>>
> > > >>>> Vito,
> > > >>>>
> > > >>>> what is the status of this KIP?
> > > >>>>
> > > >>>> I just re-read it, and have a couple of follow up comments. Why do
> > we
> > > >>>> discuss the internal exceptions you want to add? Also, do we
> really
> > > need
> > > >>>> them? Can't we just throw the correct exception directly instead
> of
> > > >>>> wrapping it later?
> > > >>>>
> > > >>>> When would we throw an `StateStoreEmptyException`? The semantics
> is
> > > >>>> unclear to me atm.
> > > >>>>
> > > >>>> Is `StreamThreadNotRunningException` really an retryable error?
> > > >>>>
> > > >>>> When the state is RUNNING, is `StateStoreClosedException` a
> > retryable
> > > >>>> error?
> > > >>>>
> > > >>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key,
> > long
> > > >>>> time); that should be added
> > > >>>>
> > > >>>>
> > > >>>> Overall I like the KIP but some details are still unclear. Maybe
> it
> > > >>>> might help if you open an PR in parallel?
> > > >>>>
> > > >>>>
> > > >>>> -Matthias
> > > >>>>
> > > >>>> On 4/24/18 8:18 AM, vito jeng wrote:
> > > >>>>> Hi, Guozhang,
> > > >>>>>
> > > >>>>> Thanks for the comment!
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> Hi, Bill,
> > > >>>>>
> > > >>>>> I'll try to make some update to make the KIP better.
> > > >>>>>
> > > >>>>> Thanks for the comment!
> > > >>>>>
> > > >>>>>
> > > >>>>> ---
> > > >>>>> Vito
> > > >>>>>
> > > >>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com>
> > > >> wrote:
> > > >>>>>
> > > >>>>>> Hi Vito,
> > > >>>>>>
> > > >>>>>> Thanks for the KIP, overall it's a +1 from me.
> > > >>>>>>
> > > >>>>>> At this point, the only thing I would change is possibly
> removing
> > > the
> > > >>>>>> listing of all methods called by the user and the listing of all
> > > store
> > > >>>>>> types and focus on what states result in which exceptions thrown
> > to
> > > >> the
> > > >>>>>> user.
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Bill
> > > >>>>>>
> > > >>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
> > wangguoz@gmail.com>
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> Thanks for the KIP Vito!
> > > >>>>>>>
> > > >>>>>>> I made a pass over the wiki and it looks great to me. I'm +1 on
> > the
> > > >>>> KIP.
> > > >>>>>>>
> > > >>>>>>> About the base class InvalidStateStoreException itself, I'd
> > > actually
> > > >>>>>>> suggest we do not deprecate it but still expose it as part of
> the
> > > >>>> public
> > > >>>>>>> API, for people who do not want to handle these cases
> differently
> > > (if
> > > >>>> we
> > > >>>>>>> deprecate it then we are enforcing them to capture all three
> > > >> exceptions
> > > >>>>>>> one-by-one).
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Guozhang
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <
> john@confluent.io
> > >
> > > >>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi Vito,
> > > >>>>>>>>
> > > >>>>>>>> Thanks for the KIP!
> > > >>>>>>>>
> > > >>>>>>>> I think it's much nicer to give callers different exceptions
> to
> > > tell
> > > >>>>>> them
> > > >>>>>>>> whether the state store got migrated, whether it's still
> > > >> initializing,
> > > >>>>>> or
> > > >>>>>>>> whether there's some unrecoverable error.
> > > >>>>>>>>
> > > >>>>>>>> In the KIP, it's typically not necessary to discuss
> > > non-user-facing
> > > >>>>>>> details
> > > >>>>>>>> such as what exceptions we will throw internally. The KIP is
> > > >> primarily
> > > >>>>>> to
> > > >>>>>>>> discuss public interface changes.
> > > >>>>>>>>
> > > >>>>>>>> You might consider simply removing all the internal details
> from
> > > the
> > > >>>>>> KIP,
> > > >>>>>>>> which will have the dual advantage that it makes the KIP
> smaller
> > > and
> > > >>>>>>> easier
> > > >>>>>>>> to agree on, as well as giving you more freedom in the
> internal
> > > >>>> details
> > > >>>>>>>> when it comes to implementation.
> > > >>>>>>>>
> > > >>>>>>>> I like your decision to have your refined exceptions extend
> > > >>>>>>>> InvalidStateStoreException to ensure backward compatibility.
> > Since
> > > >> we
> > > >>>>>>> want
> > > >>>>>>>> to encourage callers to catch the more specific exceptions,
> and
> > we
> > > >>>>>> don't
> > > >>>>>>>> expect to ever throw a raw InvalidStateStoreException anymore,
> > you
> > > >>>>>> might
> > > >>>>>>>> consider adding the @Deprecated annotation to
> > > >>>>>> InvalidStateStoreException.
> > > >>>>>>>> This will gently encourage callers to migrate to the new
> > exception
> > > >> and
> > > >>>>>>> open
> > > >>>>>>>> the possibility of removing InvalidStateStoreException
> entirely
> > > in a
> > > >>>>>>> future
> > > >>>>>>>> release.
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>> -John
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> > > >>>>>> matthias@confluent.io>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks for clarification! That makes sense to me.
> > > >>>>>>>>>
> > > >>>>>>>>> Can you update the KIP to make those suggestions explicit?
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> -Matthias
> > > >>>>>>>>>
> > > >>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> > > >>>>>>>>>> Matthias,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks for the feedback!
> > > >>>>>>>>>>
> > > >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Got it!
> > > >>>>>>>>>>
> > > >>>>>>>>>>> The (incomplete) question was, if we need
> > > >>>>>> `StateStoreFailException`
> > > >>>>>>> or
> > > >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
> > you
> > > >>>>>>> suggest
> > > >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> > anymore,
> > > >>>>>> but
> > > >>>>>>>> only
> > > >>>>>>>>>>> the new sub-classes (just to get a better understanding).
> > > >>>>>>>>>>
> > > >>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not
> thrown
> > > at
> > > >>>>>> all
> > > >>>>>>>>>> anymore,
> > > >>>>>>>>>> but only new sub-classes.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Not sure what this sentence means:
> > > >>>>>>>>>>>> The internal exception will be wrapped as category
> exception
> > > >>>>>>> finally.
> > > >>>>>>>>>>> Can you elaborate?
> > > >>>>>>>>>>
> > > >>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will
> > > throw
> > > >>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
> > > >>>>>>>>>> And then the internal exception will be wrapped as
> > > >>>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException`
> > > during
> > > >>>>>>> the
> > > >>>>>>>>>> `KafkaStreams.store()` and throw to the user.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
> > It's
> > > >>>>>>> unclear
> > > >>>>>>>>>> to me atm why they are introduced.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
> > > distinguish
> > > >>>>>>>> between
> > > >>>>>>>>>> the different kinds of InvalidStateStoreException.
> > > >>>>>>>>>> The original idea is that we can distinguish different state
> > > store
> > > >>>>>>>>>> exception for
> > > >>>>>>>>>> different handling.
> > > >>>>>>>>>> But to be honest, I am not quite sure this is necessary.
> Maybe
> > > >> have
> > > >>>>>>>>>> some change during implementation.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Does it make sense?
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> ---
> > > >>>>>>>>>> Vito
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> > > >>>>>>>> matthias@confluent.io>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Thanks for the update Vito!
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> The (incomplete) question was, if we need
> > > >>>>>> `StateStoreFailException`
> > > >>>>>>> or
> > > >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
> > you
> > > >>>>>>> suggest
> > > >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> > anymore,
> > > >>>>>> but
> > > >>>>>>>> only
> > > >>>>>>>>>>> the new sub-classes (just to get a better understanding).
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Not sure what this sentence means:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> The internal exception will be wrapped as category
> exception
> > > >>>>>>> finally.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Can you elaborate?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
> > It's
> > > >>>>>>> unclear
> > > >>>>>>>>>>> to me atm why they are introduced.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> -Matthias
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> > > >>>>>>>>>>>> Matthias,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Thanks for the review.
> > > >>>>>>>>>>>> I reply separately in the following sections.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ---
> > > >>>>>>>>>>>> Vito
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> > > >>>>>>>> matthias@confluent.io
> > > >>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long
> pause...
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Seems you did a very thorough investigation of the code.
> > It's
> > > >>>>>>> useful
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>> understand what user facing interfaces are affected.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I also think too detailed. Especially the section `Changes
> > in
> > > >>>>>> call
> > > >>>>>>>>>>> trace`.
> > > >>>>>>>>>>>> Do you think it should be removed?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> To summarize my current understanding of your KIP, the
> main
> > > >>>>>> change
> > > >>>>>>>> is
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>> introduce new exceptions that extend
> > > >>>>>> `InvalidStateStoreException`.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
> > > >>>>>>>>>>>> I think the best way is that all new exceptions extend
> from
> > > >>>>>>>>>>>> `InvalidStateStoreException`.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Some questions:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException`
> > be
> > > >>>>>> used
> > > >>>>>>>> for
> > > >>>>>>>>>>>>> this purpose?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Does this question miss some word?
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
> > > >>>>>>>> ngException`
> > > >>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> > > >>>>>>>>>>> `StateStoreFailException`
> > > >>>>>>>>>>>>> ?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> I apologize for not well written parts. I tried to modify
> > some
> > > >>>>>> code
> > > >>>>>>>> in
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>> recent period and modify the KIP.
> > > >>>>>>>>>>>> The modification is now complete. Please look again.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> -Matthias
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> > > >>>>>>>>>>>>>> Matthias,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Sorry for not response these days.
> > > >>>>>>>>>>>>>> I just finished it. Please have a look. :)
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> ---
> > > >>>>>>>>>>>>>> Vito
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> > > >>>>>>>>>>> matthias@confluent.io>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Is there any update on this KIP?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> -Matthias
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> > > >>>>>>>>>>>>>>>> Matthias,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thank you for your response.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I think you are right. We need to look at the state
> both
> > > of
> > > >>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread and
> > > state
> > > >>>>>>>> store,
> > > >>>>>>>>>>>>>>>> I am currently rewriting the KIP.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> ---
> > > >>>>>>>>>>>>>>>> Vito
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> > > >>>>>>>>>>>>> matthias@confluent.io>
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Vito,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Sorry for this late reply.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> There can be two cases:
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not
> > > hosted
> > > >>>>>> an
> > > >>>>>>>> the
> > > >>>>>>>>>>>>>>>>> application instance anymore,
> > > >>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
> > > >>>>>>> rebalance
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> For the first case, users need to rediscover the
> store.
> > > For
> > > >>>>>>> the
> > > >>>>>>>>>>> second
> > > >>>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN,
> or
> > > >>>>>>>>> NOT_RUNNING,
> > > >>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
> > rediscover
> > > or
> > > >>>>>>>> retry.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Does this make sense?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> -Matthias
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> > > >>>>>>>>>>>>>>>>>> Matthias,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> I try to clarify some concept.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user
> > can
> > > >>>>>> just
> > > >>>>>>>>> plain
> > > >>>>>>>>>>>>>>>>> retry.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> > > >>>>>>> NOT_RUNNING,
> > > >>>>>>>>> it
> > > >>>>>>>>>>>>>>> means
> > > >>>>>>>>>>>>>>>>>> state store migrated to another instance, the user
> > needs
> > > >> to
> > > >>>>>>>>>>>>> rediscover
> > > >>>>>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>> store.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Is my understanding correct?
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> ---
> > > >>>>>>>>>>>>>>>>>> Vito
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> > > >>>>>>>>>>>>>>> matthias@confluent.io>
> > > >>>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea
> of
> > > the
> > > >>>>>>> Jira
> > > >>>>>>>>>>> was,
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
> > > >>>>>>> strategies
> > > >>>>>>>> to
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>> user.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a
> > user
> > > >>>>>> just
> > > >>>>>>>> need
> > > >>>>>>>>>>> to
> > > >>>>>>>>>>>>>>> wait
> > > >>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand,
> if
> > a
> > > >>>>>> store
> > > >>>>>>>> go
> > > >>>>>>>>>>>>>>> migrated
> > > >>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the
> > > store
> > > >>>>>>>>> instead
> > > >>>>>>>>>>>>> of a
> > > >>>>>>>>>>>>>>>>>>> "plain retry".
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Not sure if there is something else?
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks
> > > about
> > > >>>>>>> this
> > > >>>>>>>>>>> ideas
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>> reasoning.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> -Matthias
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > > >>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure
> > out
> > > >>>>>> first
> > > >>>>>>>> is
> > > >>>>>>>>>>> what
> > > >>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the users,
> if
> > > we
> > > >>>>>>> only
> > > >>>>>>>>>>> wants
> > > >>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we
> > > should
> > > >>>>>>>> rename
> > > >>>>>>>>>>> the
> > > >>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> > > >>>>>>>>> StateStoreClosedException
> > > >>>>>>>>>>>>>>>>> classes.
> > > >>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
> > > possible
> > > >>>>>>>>> internal
> > > >>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call
> > > trace,
> > > >>>>>> and
> > > >>>>>>>>> which
> > > >>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and
> > which
> > > >>>>>> ones
> > > >>>>>>>>>>> should
> > > >>>>>>>>>>>>> be
> > > >>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should
> > not
> > > >> be
> > > >>>>>>>>> wrapped
> > > >>>>>>>>>>>>> at
> > > >>>>>>>>>>>>>>>>> all
> > > >>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> Guozhang
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> > > >>>>>>>>> vito@is-land.com.tw>
> > > >>>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
> > > confluence/display/KAFKA/KIP-
> > > >>>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> > > >>>>>>>>>>> different+errors
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> Please have a look.
> > > >>>>>>>>>>>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>> ---
> > > >>>>>>>>>>>>>>>>>>>>> Vito
> > > >>>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>> -- Guozhang
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by John Roesler <jo...@confluent.io>.
Hi Vito,
I'm glad to hear you're well again!
I didn't follow the addition of a new method to the QueryableStoreType
interface. Can you elaborate why this is necessary to support the new
exception types?
Also, looking over your KIP again, it seems valuable to introduce
"retriable store exception" and "fatal store exception" marker interfaces
that the various exceptions can mix in. It would be nice from a usability
perspective to be able to just log and retry on any "retriable" exception
and log and shutdown on any fatal exception.
Thanks,
-John
On Fri, Oct 5, 2018 at 11:47 AM Guozhang Wang <wa...@gmail.com> wrote:
> Thanks for the explanation, that makes sense.
>
>
> Guozhang
>
>
> On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > The scenario I had I mind was, that KS is started in one thread while a
> > second thread has a reference to the object to issue queries.
> >
> > If a query is issue before the "main thread" started KS, and the "query
> > thread" knows that it will eventually get started, it can retry. On the
> > other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is impossible
> > to issue any query against it now or in the future and thus the error is
> > not retryable.
> >
> >
> > -Matthias
> >
> > On 6/25/18 10:15 AM, Guozhang Wang wrote:
> > > I'm wondering if StreamThreadNotStarted could be merged into
> > > StreamThreadNotRunning, because I think users' handling logic for the
> > third
> > > case would be likely the same as the second. Do you have some scenarios
> > > where users may want to handle them differently?
> > >
> > > Guozhang
> > >
> > > On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <
> matthias@confluent.io>
> > > wrote:
> > >
> > >> Sorry to hear! Get well soon!
> > >>
> > >> It's not a big deal if the KIP stalls a little bit. Feel free to pick
> it
> > >> up again when you find time.
> > >>
> > >>>>> Is `StreamThreadNotRunningException` really an retryable error?
> > >>>>
> > >>>> When KafkaStream state is REBALANCING, I think it is a retryable
> > error.
> > >>>>
> > >>>> StreamThreadStateStoreProvider#stores() will throw
> > >>>> StreamThreadNotRunningException when StreamThread state is not
> > >> RUNNING. The
> > >>>> user can retry until KafkaStream state is RUNNING.
> > >>
> > >> I see. If this is the intention, than I would suggest to have two (or
> > >> maybe three) different exceptions:
> > >>
> > >> - StreamThreadRebalancingException (retryable)
> > >> - StreamThreadNotRunning (not retryable -- thrown if in state
> > >> PENDING_SHUTDOWN or DEAD
> > >> - maybe StreamThreadNotStarted (for state CREATED)
> > >>
> > >> The last one is tricky and could also be merged into one of the first
> > >> two, depending if you want to argue that it's retryable or not. (Just
> > >> food for though -- not sure what others think.)
> > >>
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 6/22/18 8:06 AM, vito jeng wrote:
> > >>> Matthias,
> > >>>
> > >>> Thank you for your assistance.
> > >>>
> > >>>> what is the status of this KIP?
> > >>>
> > >>> Unfortunately, there is no further progress.
> > >>> About seven weeks ago, I was injured in sports. I had a broken wrist
> on
> > >>> my left wrist.
> > >>> Many jobs are affected, including this KIP and implementation.
> > >>>
> > >>>
> > >>>> I just re-read it, and have a couple of follow up comments. Why do
> we
> > >>>> discuss the internal exceptions you want to add? Also, do we really
> > need
> > >>>> them? Can't we just throw the correct exception directly instead of
> > >>>> wrapping it later?
> > >>>
> > >>> I think you may be right. As I say in the previous:
> > >>> "The original idea is that we can distinguish different state store
> > >>> exception for different handling. But to be honest, I am not quite
> sure
> > >>> this is necessary. Maybe have some change during implementation."
> > >>>
> > >>> During the implementation, I also feel we maybe not need wrapper it.
> > >>> We can just throw the correctly directly.
> > >>>
> > >>>
> > >>>> Is `StreamThreadNotRunningException` really an retryable error?
> > >>>
> > >>> When KafkaStream state is REBALANCING, I think it is a retryable
> error.
> > >>>
> > >>> StreamThreadStateStoreProvider#stores() will throw
> > >>> StreamThreadNotRunningException when StreamThread state is not
> > RUNNING.
> > >> The
> > >>> user can retry until KafkaStream state is RUNNING.
> > >>>
> > >>>
> > >>>> When would we throw an `StateStoreEmptyException`? The semantics is
> > >>> unclear to me atm.
> > >>>
> > >>>> When the state is RUNNING, is `StateStoreClosedException` a
> retryable
> > >>> error?
> > >>>
> > >>> These two comments will be answered in another mail.
> > >>>
> > >>>
> > >>>
> > >>> ---
> > >>> Vito
> > >>>
> > >>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
> > matthias@confluent.io>
> > >>> wrote:
> > >>>
> > >>>> Vito,
> > >>>>
> > >>>> what is the status of this KIP?
> > >>>>
> > >>>> I just re-read it, and have a couple of follow up comments. Why do
> we
> > >>>> discuss the internal exceptions you want to add? Also, do we really
> > need
> > >>>> them? Can't we just throw the correct exception directly instead of
> > >>>> wrapping it later?
> > >>>>
> > >>>> When would we throw an `StateStoreEmptyException`? The semantics is
> > >>>> unclear to me atm.
> > >>>>
> > >>>> Is `StreamThreadNotRunningException` really an retryable error?
> > >>>>
> > >>>> When the state is RUNNING, is `StateStoreClosedException` a
> retryable
> > >>>> error?
> > >>>>
> > >>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key,
> long
> > >>>> time); that should be added
> > >>>>
> > >>>>
> > >>>> Overall I like the KIP but some details are still unclear. Maybe it
> > >>>> might help if you open an PR in parallel?
> > >>>>
> > >>>>
> > >>>> -Matthias
> > >>>>
> > >>>> On 4/24/18 8:18 AM, vito jeng wrote:
> > >>>>> Hi, Guozhang,
> > >>>>>
> > >>>>> Thanks for the comment!
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Hi, Bill,
> > >>>>>
> > >>>>> I'll try to make some update to make the KIP better.
> > >>>>>
> > >>>>> Thanks for the comment!
> > >>>>>
> > >>>>>
> > >>>>> ---
> > >>>>> Vito
> > >>>>>
> > >>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com>
> > >> wrote:
> > >>>>>
> > >>>>>> Hi Vito,
> > >>>>>>
> > >>>>>> Thanks for the KIP, overall it's a +1 from me.
> > >>>>>>
> > >>>>>> At this point, the only thing I would change is possibly removing
> > the
> > >>>>>> listing of all methods called by the user and the listing of all
> > store
> > >>>>>> types and focus on what states result in which exceptions thrown
> to
> > >> the
> > >>>>>> user.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Bill
> > >>>>>>
> > >>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <
> wangguoz@gmail.com>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks for the KIP Vito!
> > >>>>>>>
> > >>>>>>> I made a pass over the wiki and it looks great to me. I'm +1 on
> the
> > >>>> KIP.
> > >>>>>>>
> > >>>>>>> About the base class InvalidStateStoreException itself, I'd
> > actually
> > >>>>>>> suggest we do not deprecate it but still expose it as part of the
> > >>>> public
> > >>>>>>> API, for people who do not want to handle these cases differently
> > (if
> > >>>> we
> > >>>>>>> deprecate it then we are enforcing them to capture all three
> > >> exceptions
> > >>>>>>> one-by-one).
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Guozhang
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <john@confluent.io
> >
> > >>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Vito,
> > >>>>>>>>
> > >>>>>>>> Thanks for the KIP!
> > >>>>>>>>
> > >>>>>>>> I think it's much nicer to give callers different exceptions to
> > tell
> > >>>>>> them
> > >>>>>>>> whether the state store got migrated, whether it's still
> > >> initializing,
> > >>>>>> or
> > >>>>>>>> whether there's some unrecoverable error.
> > >>>>>>>>
> > >>>>>>>> In the KIP, it's typically not necessary to discuss
> > non-user-facing
> > >>>>>>> details
> > >>>>>>>> such as what exceptions we will throw internally. The KIP is
> > >> primarily
> > >>>>>> to
> > >>>>>>>> discuss public interface changes.
> > >>>>>>>>
> > >>>>>>>> You might consider simply removing all the internal details from
> > the
> > >>>>>> KIP,
> > >>>>>>>> which will have the dual advantage that it makes the KIP smaller
> > and
> > >>>>>>> easier
> > >>>>>>>> to agree on, as well as giving you more freedom in the internal
> > >>>> details
> > >>>>>>>> when it comes to implementation.
> > >>>>>>>>
> > >>>>>>>> I like your decision to have your refined exceptions extend
> > >>>>>>>> InvalidStateStoreException to ensure backward compatibility.
> Since
> > >> we
> > >>>>>>> want
> > >>>>>>>> to encourage callers to catch the more specific exceptions, and
> we
> > >>>>>> don't
> > >>>>>>>> expect to ever throw a raw InvalidStateStoreException anymore,
> you
> > >>>>>> might
> > >>>>>>>> consider adding the @Deprecated annotation to
> > >>>>>> InvalidStateStoreException.
> > >>>>>>>> This will gently encourage callers to migrate to the new
> exception
> > >> and
> > >>>>>>> open
> > >>>>>>>> the possibility of removing InvalidStateStoreException entirely
> > in a
> > >>>>>>> future
> > >>>>>>>> release.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> -John
> > >>>>>>>>
> > >>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> > >>>>>> matthias@confluent.io>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks for clarification! That makes sense to me.
> > >>>>>>>>>
> > >>>>>>>>> Can you update the KIP to make those suggestions explicit?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> -Matthias
> > >>>>>>>>>
> > >>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> > >>>>>>>>>> Matthias,
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks for the feedback!
> > >>>>>>>>>>
> > >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
> > >>>>>>>>>>
> > >>>>>>>>>> Got it!
> > >>>>>>>>>>
> > >>>>>>>>>>> The (incomplete) question was, if we need
> > >>>>>> `StateStoreFailException`
> > >>>>>>> or
> > >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
> you
> > >>>>>>> suggest
> > >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> anymore,
> > >>>>>> but
> > >>>>>>>> only
> > >>>>>>>>>>> the new sub-classes (just to get a better understanding).
> > >>>>>>>>>>
> > >>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown
> > at
> > >>>>>> all
> > >>>>>>>>>> anymore,
> > >>>>>>>>>> but only new sub-classes.
> > >>>>>>>>>>
> > >>>>>>>>>>> Not sure what this sentence means:
> > >>>>>>>>>>>> The internal exception will be wrapped as category exception
> > >>>>>>> finally.
> > >>>>>>>>>>> Can you elaborate?
> > >>>>>>>>>>
> > >>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will
> > throw
> > >>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
> > >>>>>>>>>> And then the internal exception will be wrapped as
> > >>>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException`
> > during
> > >>>>>>> the
> > >>>>>>>>>> `KafkaStreams.store()` and throw to the user.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
> It's
> > >>>>>>> unclear
> > >>>>>>>>>> to me atm why they are introduced.
> > >>>>>>>>>>
> > >>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
> > distinguish
> > >>>>>>>> between
> > >>>>>>>>>> the different kinds of InvalidStateStoreException.
> > >>>>>>>>>> The original idea is that we can distinguish different state
> > store
> > >>>>>>>>>> exception for
> > >>>>>>>>>> different handling.
> > >>>>>>>>>> But to be honest, I am not quite sure this is necessary. Maybe
> > >> have
> > >>>>>>>>>> some change during implementation.
> > >>>>>>>>>>
> > >>>>>>>>>> Does it make sense?
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> ---
> > >>>>>>>>>> Vito
> > >>>>>>>>>>
> > >>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> > >>>>>>>> matthias@confluent.io>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Thanks for the update Vito!
> > >>>>>>>>>>>
> > >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> The (incomplete) question was, if we need
> > >>>>>> `StateStoreFailException`
> > >>>>>>> or
> > >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do
> you
> > >>>>>>> suggest
> > >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all
> anymore,
> > >>>>>> but
> > >>>>>>>> only
> > >>>>>>>>>>> the new sub-classes (just to get a better understanding).
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Not sure what this sentence means:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> The internal exception will be wrapped as category exception
> > >>>>>>> finally.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Can you elaborate?
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Can you explain the purpose of the "internal exceptions".
> It's
> > >>>>>>> unclear
> > >>>>>>>>>>> to me atm why they are introduced.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> -Matthias
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> > >>>>>>>>>>>> Matthias,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Thanks for the review.
> > >>>>>>>>>>>> I reply separately in the following sections.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> Vito
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> > >>>>>>>> matthias@confluent.io
> > >>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Seems you did a very thorough investigation of the code.
> It's
> > >>>>>>> useful
> > >>>>>>>>> to
> > >>>>>>>>>>>>> understand what user facing interfaces are affected.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I also think too detailed. Especially the section `Changes
> in
> > >>>>>> call
> > >>>>>>>>>>> trace`.
> > >>>>>>>>>>>> Do you think it should be removed?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> To summarize my current understanding of your KIP, the main
> > >>>>>> change
> > >>>>>>>> is
> > >>>>>>>>> to
> > >>>>>>>>>>>>> introduce new exceptions that extend
> > >>>>>> `InvalidStateStoreException`.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
> > >>>>>>>>>>>> I think the best way is that all new exceptions extend from
> > >>>>>>>>>>>> `InvalidStateStoreException`.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Some questions:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException`
> be
> > >>>>>> used
> > >>>>>>>> for
> > >>>>>>>>>>>>> this purpose?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Does this question miss some word?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
> > >>>>>>>> ngException`
> > >>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> > >>>>>>>>>>> `StateStoreFailException`
> > >>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> I apologize for not well written parts. I tried to modify
> some
> > >>>>>> code
> > >>>>>>>> in
> > >>>>>>>>>>> the
> > >>>>>>>>>>>> recent period and modify the KIP.
> > >>>>>>>>>>>> The modification is now complete. Please look again.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> -Matthias
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> > >>>>>>>>>>>>>> Matthias,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Sorry for not response these days.
> > >>>>>>>>>>>>>> I just finished it. Please have a look. :)
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>> Vito
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> > >>>>>>>>>>> matthias@confluent.io>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Is there any update on this KIP?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> -Matthias
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> > >>>>>>>>>>>>>>>> Matthias,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thank you for your response.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I think you are right. We need to look at the state both
> > of
> > >>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread and
> > state
> > >>>>>>>> store,
> > >>>>>>>>>>>>>>>> I am currently rewriting the KIP.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>>> Vito
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> > >>>>>>>>>>>>> matthias@confluent.io>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Vito,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Sorry for this late reply.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> There can be two cases:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not
> > hosted
> > >>>>>> an
> > >>>>>>>> the
> > >>>>>>>>>>>>>>>>> application instance anymore,
> > >>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
> > >>>>>>> rebalance
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> For the first case, users need to rediscover the store.
> > For
> > >>>>>>> the
> > >>>>>>>>>>> second
> > >>>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> > >>>>>>>>> NOT_RUNNING,
> > >>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot
> rediscover
> > or
> > >>>>>>>> retry.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Does this make sense?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> -Matthias
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> > >>>>>>>>>>>>>>>>>> Matthias,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I try to clarify some concept.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user
> can
> > >>>>>> just
> > >>>>>>>>> plain
> > >>>>>>>>>>>>>>>>> retry.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> > >>>>>>> NOT_RUNNING,
> > >>>>>>>>> it
> > >>>>>>>>>>>>>>> means
> > >>>>>>>>>>>>>>>>>> state store migrated to another instance, the user
> needs
> > >> to
> > >>>>>>>>>>>>> rediscover
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> store.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Is my understanding correct?
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>>>>> Vito
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> > >>>>>>>>>>>>>>> matthias@confluent.io>
> > >>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of
> > the
> > >>>>>>> Jira
> > >>>>>>>>>>> was,
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
> > >>>>>>> strategies
> > >>>>>>>> to
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>> user.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a
> user
> > >>>>>> just
> > >>>>>>>> need
> > >>>>>>>>>>> to
> > >>>>>>>>>>>>>>> wait
> > >>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand, if
> a
> > >>>>>> store
> > >>>>>>>> go
> > >>>>>>>>>>>>>>> migrated
> > >>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the
> > store
> > >>>>>>>>> instead
> > >>>>>>>>>>>>> of a
> > >>>>>>>>>>>>>>>>>>> "plain retry".
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Not sure if there is something else?
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks
> > about
> > >>>>>>> this
> > >>>>>>>>>>> ideas
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> reasoning.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> -Matthias
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > >>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure
> out
> > >>>>>> first
> > >>>>>>>> is
> > >>>>>>>>>>> what
> > >>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the users, if
> > we
> > >>>>>>> only
> > >>>>>>>>>>> wants
> > >>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we
> > should
> > >>>>>>>> rename
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> > >>>>>>>>> StateStoreClosedException
> > >>>>>>>>>>>>>>>>> classes.
> > >>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
> > possible
> > >>>>>>>>> internal
> > >>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call
> > trace,
> > >>>>>> and
> > >>>>>>>>> which
> > >>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and
> which
> > >>>>>> ones
> > >>>>>>>>>>> should
> > >>>>>>>>>>>>> be
> > >>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should
> not
> > >> be
> > >>>>>>>>> wrapped
> > >>>>>>>>>>>>> at
> > >>>>>>>>>>>>>>>>> all
> > >>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> Guozhang
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> > >>>>>>>>> vito@is-land.com.tw>
> > >>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-
> > >>>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> > >>>>>>>>>>> different+errors
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> Please have a look.
> > >>>>>>>>>>>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>> ---
> > >>>>>>>>>>>>>>>>>>>>> Vito
> > >>>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>> -- Guozhang
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >
> > >
> >
> >
>
>
> --
> -- Guozhang
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Guozhang Wang <wa...@gmail.com>.
Thanks for the explanation, that makes sense.
Guozhang
On Mon, Jun 25, 2018 at 2:28 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> The scenario I had I mind was, that KS is started in one thread while a
> second thread has a reference to the object to issue queries.
>
> If a query is issue before the "main thread" started KS, and the "query
> thread" knows that it will eventually get started, it can retry. On the
> other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is impossible
> to issue any query against it now or in the future and thus the error is
> not retryable.
>
>
> -Matthias
>
> On 6/25/18 10:15 AM, Guozhang Wang wrote:
> > I'm wondering if StreamThreadNotStarted could be merged into
> > StreamThreadNotRunning, because I think users' handling logic for the
> third
> > case would be likely the same as the second. Do you have some scenarios
> > where users may want to handle them differently?
> >
> > Guozhang
> >
> > On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Sorry to hear! Get well soon!
> >>
> >> It's not a big deal if the KIP stalls a little bit. Feel free to pick it
> >> up again when you find time.
> >>
> >>>>> Is `StreamThreadNotRunningException` really an retryable error?
> >>>>
> >>>> When KafkaStream state is REBALANCING, I think it is a retryable
> error.
> >>>>
> >>>> StreamThreadStateStoreProvider#stores() will throw
> >>>> StreamThreadNotRunningException when StreamThread state is not
> >> RUNNING. The
> >>>> user can retry until KafkaStream state is RUNNING.
> >>
> >> I see. If this is the intention, than I would suggest to have two (or
> >> maybe three) different exceptions:
> >>
> >> - StreamThreadRebalancingException (retryable)
> >> - StreamThreadNotRunning (not retryable -- thrown if in state
> >> PENDING_SHUTDOWN or DEAD
> >> - maybe StreamThreadNotStarted (for state CREATED)
> >>
> >> The last one is tricky and could also be merged into one of the first
> >> two, depending if you want to argue that it's retryable or not. (Just
> >> food for though -- not sure what others think.)
> >>
> >>
> >>
> >> -Matthias
> >>
> >> On 6/22/18 8:06 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Thank you for your assistance.
> >>>
> >>>> what is the status of this KIP?
> >>>
> >>> Unfortunately, there is no further progress.
> >>> About seven weeks ago, I was injured in sports. I had a broken wrist on
> >>> my left wrist.
> >>> Many jobs are affected, including this KIP and implementation.
> >>>
> >>>
> >>>> I just re-read it, and have a couple of follow up comments. Why do we
> >>>> discuss the internal exceptions you want to add? Also, do we really
> need
> >>>> them? Can't we just throw the correct exception directly instead of
> >>>> wrapping it later?
> >>>
> >>> I think you may be right. As I say in the previous:
> >>> "The original idea is that we can distinguish different state store
> >>> exception for different handling. But to be honest, I am not quite sure
> >>> this is necessary. Maybe have some change during implementation."
> >>>
> >>> During the implementation, I also feel we maybe not need wrapper it.
> >>> We can just throw the correctly directly.
> >>>
> >>>
> >>>> Is `StreamThreadNotRunningException` really an retryable error?
> >>>
> >>> When KafkaStream state is REBALANCING, I think it is a retryable error.
> >>>
> >>> StreamThreadStateStoreProvider#stores() will throw
> >>> StreamThreadNotRunningException when StreamThread state is not
> RUNNING.
> >> The
> >>> user can retry until KafkaStream state is RUNNING.
> >>>
> >>>
> >>>> When would we throw an `StateStoreEmptyException`? The semantics is
> >>> unclear to me atm.
> >>>
> >>>> When the state is RUNNING, is `StateStoreClosedException` a retryable
> >>> error?
> >>>
> >>> These two comments will be answered in another mail.
> >>>
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <
> matthias@confluent.io>
> >>> wrote:
> >>>
> >>>> Vito,
> >>>>
> >>>> what is the status of this KIP?
> >>>>
> >>>> I just re-read it, and have a couple of follow up comments. Why do we
> >>>> discuss the internal exceptions you want to add? Also, do we really
> need
> >>>> them? Can't we just throw the correct exception directly instead of
> >>>> wrapping it later?
> >>>>
> >>>> When would we throw an `StateStoreEmptyException`? The semantics is
> >>>> unclear to me atm.
> >>>>
> >>>> Is `StreamThreadNotRunningException` really an retryable error?
> >>>>
> >>>> When the state is RUNNING, is `StateStoreClosedException` a retryable
> >>>> error?
> >>>>
> >>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
> >>>> time); that should be added
> >>>>
> >>>>
> >>>> Overall I like the KIP but some details are still unclear. Maybe it
> >>>> might help if you open an PR in parallel?
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 4/24/18 8:18 AM, vito jeng wrote:
> >>>>> Hi, Guozhang,
> >>>>>
> >>>>> Thanks for the comment!
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hi, Bill,
> >>>>>
> >>>>> I'll try to make some update to make the KIP better.
> >>>>>
> >>>>> Thanks for the comment!
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> Hi Vito,
> >>>>>>
> >>>>>> Thanks for the KIP, overall it's a +1 from me.
> >>>>>>
> >>>>>> At this point, the only thing I would change is possibly removing
> the
> >>>>>> listing of all methods called by the user and the listing of all
> store
> >>>>>> types and focus on what states result in which exceptions thrown to
> >> the
> >>>>>> user.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Bill
> >>>>>>
> >>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> Thanks for the KIP Vito!
> >>>>>>>
> >>>>>>> I made a pass over the wiki and it looks great to me. I'm +1 on the
> >>>> KIP.
> >>>>>>>
> >>>>>>> About the base class InvalidStateStoreException itself, I'd
> actually
> >>>>>>> suggest we do not deprecate it but still expose it as part of the
> >>>> public
> >>>>>>> API, for people who do not want to handle these cases differently
> (if
> >>>> we
> >>>>>>> deprecate it then we are enforcing them to capture all three
> >> exceptions
> >>>>>>> one-by-one).
> >>>>>>>
> >>>>>>>
> >>>>>>> Guozhang
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Vito,
> >>>>>>>>
> >>>>>>>> Thanks for the KIP!
> >>>>>>>>
> >>>>>>>> I think it's much nicer to give callers different exceptions to
> tell
> >>>>>> them
> >>>>>>>> whether the state store got migrated, whether it's still
> >> initializing,
> >>>>>> or
> >>>>>>>> whether there's some unrecoverable error.
> >>>>>>>>
> >>>>>>>> In the KIP, it's typically not necessary to discuss
> non-user-facing
> >>>>>>> details
> >>>>>>>> such as what exceptions we will throw internally. The KIP is
> >> primarily
> >>>>>> to
> >>>>>>>> discuss public interface changes.
> >>>>>>>>
> >>>>>>>> You might consider simply removing all the internal details from
> the
> >>>>>> KIP,
> >>>>>>>> which will have the dual advantage that it makes the KIP smaller
> and
> >>>>>>> easier
> >>>>>>>> to agree on, as well as giving you more freedom in the internal
> >>>> details
> >>>>>>>> when it comes to implementation.
> >>>>>>>>
> >>>>>>>> I like your decision to have your refined exceptions extend
> >>>>>>>> InvalidStateStoreException to ensure backward compatibility. Since
> >> we
> >>>>>>> want
> >>>>>>>> to encourage callers to catch the more specific exceptions, and we
> >>>>>> don't
> >>>>>>>> expect to ever throw a raw InvalidStateStoreException anymore, you
> >>>>>> might
> >>>>>>>> consider adding the @Deprecated annotation to
> >>>>>> InvalidStateStoreException.
> >>>>>>>> This will gently encourage callers to migrate to the new exception
> >> and
> >>>>>>> open
> >>>>>>>> the possibility of removing InvalidStateStoreException entirely
> in a
> >>>>>>> future
> >>>>>>>> release.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> -John
> >>>>>>>>
> >>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >>>>>> matthias@confluent.io>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks for clarification! That makes sense to me.
> >>>>>>>>>
> >>>>>>>>> Can you update the KIP to make those suggestions explicit?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>>>>>> Matthias,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the feedback!
> >>>>>>>>>>
> >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>>>>>
> >>>>>>>>>> Got it!
> >>>>>>>>>>
> >>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>> `StateStoreFailException`
> >>>>>>> or
> >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>>>>>> suggest
> >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >>>>>> but
> >>>>>>>> only
> >>>>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>>>
> >>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown
> at
> >>>>>> all
> >>>>>>>>>> anymore,
> >>>>>>>>>> but only new sub-classes.
> >>>>>>>>>>
> >>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>> The internal exception will be wrapped as category exception
> >>>>>>> finally.
> >>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>
> >>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will
> throw
> >>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
> >>>>>>>>>> And then the internal exception will be wrapped as
> >>>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException`
> during
> >>>>>>> the
> >>>>>>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>>>>>> unclear
> >>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>
> >>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to
> distinguish
> >>>>>>>> between
> >>>>>>>>>> the different kinds of InvalidStateStoreException.
> >>>>>>>>>> The original idea is that we can distinguish different state
> store
> >>>>>>>>>> exception for
> >>>>>>>>>> different handling.
> >>>>>>>>>> But to be honest, I am not quite sure this is necessary. Maybe
> >> have
> >>>>>>>>>> some change during implementation.
> >>>>>>>>>>
> >>>>>>>>>> Does it make sense?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> Vito
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>>>>>> matthias@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks for the update Vito!
> >>>>>>>>>>>
> >>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> The (incomplete) question was, if we need
> >>>>>> `StateStoreFailException`
> >>>>>>> or
> >>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>>>>>> suggest
> >>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >>>>>> but
> >>>>>>>> only
> >>>>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>>>
> >>>>>>>>>>>> The internal exception will be wrapped as category exception
> >>>>>>> finally.
> >>>>>>>>>>>
> >>>>>>>>>>> Can you elaborate?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>>>>>> unclear
> >>>>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -Matthias
> >>>>>>>>>>>
> >>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the review.
> >>>>>>>>>>>> I reply separately in the following sections.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>>>>>> matthias@confluent.io
> >>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Seems you did a very thorough investigation of the code. It's
> >>>>>>> useful
> >>>>>>>>> to
> >>>>>>>>>>>>> understand what user facing interfaces are affected.
> >>>>>>>>>>>>
> >>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> I also think too detailed. Especially the section `Changes in
> >>>>>> call
> >>>>>>>>>>> trace`.
> >>>>>>>>>>>> Do you think it should be removed?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> To summarize my current understanding of your KIP, the main
> >>>>>> change
> >>>>>>>> is
> >>>>>>>>> to
> >>>>>>>>>>>>> introduce new exceptions that extend
> >>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
> >>>>>>>>>>>> I think the best way is that all new exceptions extend from
> >>>>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Some questions:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException` be
> >>>>>> used
> >>>>>>>> for
> >>>>>>>>>>>>> this purpose?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Does this question miss some word?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
> >>>>>>>> ngException`
> >>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>>>> ?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>> I apologize for not well written parts. I tried to modify some
> >>>>>> code
> >>>>>>>> in
> >>>>>>>>>>> the
> >>>>>>>>>>>> recent period and modify the KIP.
> >>>>>>>>>>>> The modification is now complete. Please look again.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I think you are right. We need to look at the state both
> of
> >>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread and
> state
> >>>>>>>> store,
> >>>>>>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not
> hosted
> >>>>>> an
> >>>>>>>> the
> >>>>>>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
> >>>>>>> rebalance
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For the first case, users need to rediscover the store.
> For
> >>>>>>> the
> >>>>>>>>>>> second
> >>>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> >>>>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot rediscover
> or
> >>>>>>>> retry.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user can
> >>>>>> just
> >>>>>>>>> plain
> >>>>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> >>>>>>> NOT_RUNNING,
> >>>>>>>>> it
> >>>>>>>>>>>>>>> means
> >>>>>>>>>>>>>>>>>> state store migrated to another instance, the user needs
> >> to
> >>>>>>>>>>>>> rediscover
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of
> the
> >>>>>>> Jira
> >>>>>>>>>>> was,
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
> >>>>>>> strategies
> >>>>>>>> to
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a user
> >>>>>> just
> >>>>>>>> need
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>> wait
> >>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand, if a
> >>>>>> store
> >>>>>>>> go
> >>>>>>>>>>>>>>> migrated
> >>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the
> store
> >>>>>>>>> instead
> >>>>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks
> about
> >>>>>>> this
> >>>>>>>>>>> ideas
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
> >>>>>> first
> >>>>>>>> is
> >>>>>>>>>>> what
> >>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the users, if
> we
> >>>>>>> only
> >>>>>>>>>>> wants
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we
> should
> >>>>>>>> rename
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>>>>>> StateStoreClosedException
> >>>>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>>>> And then from there we should list what are the
> possible
> >>>>>>>>> internal
> >>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call
> trace,
> >>>>>> and
> >>>>>>>>> which
> >>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and which
> >>>>>> ones
> >>>>>>>>>>> should
> >>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should not
> >> be
> >>>>>>>>> wrapped
> >>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> >>>>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>>>>>> different+errors
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> -- Guozhang
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
> >
>
>
--
-- Guozhang
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
The scenario I had I mind was, that KS is started in one thread while a
second thread has a reference to the object to issue queries.
If a query is issue before the "main thread" started KS, and the "query
thread" knows that it will eventually get started, it can retry. On the
other hand, if KS is in state PENDING_SHUTDOWN or DEAD, it is impossible
to issue any query against it now or in the future and thus the error is
not retryable.
-Matthias
On 6/25/18 10:15 AM, Guozhang Wang wrote:
> I'm wondering if StreamThreadNotStarted could be merged into
> StreamThreadNotRunning, because I think users' handling logic for the third
> case would be likely the same as the second. Do you have some scenarios
> where users may want to handle them differently?
>
> Guozhang
>
> On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Sorry to hear! Get well soon!
>>
>> It's not a big deal if the KIP stalls a little bit. Feel free to pick it
>> up again when you find time.
>>
>>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>
>>>> When KafkaStream state is REBALANCING, I think it is a retryable error.
>>>>
>>>> StreamThreadStateStoreProvider#stores() will throw
>>>> StreamThreadNotRunningException when StreamThread state is not
>> RUNNING. The
>>>> user can retry until KafkaStream state is RUNNING.
>>
>> I see. If this is the intention, than I would suggest to have two (or
>> maybe three) different exceptions:
>>
>> - StreamThreadRebalancingException (retryable)
>> - StreamThreadNotRunning (not retryable -- thrown if in state
>> PENDING_SHUTDOWN or DEAD
>> - maybe StreamThreadNotStarted (for state CREATED)
>>
>> The last one is tricky and could also be merged into one of the first
>> two, depending if you want to argue that it's retryable or not. (Just
>> food for though -- not sure what others think.)
>>
>>
>>
>> -Matthias
>>
>> On 6/22/18 8:06 AM, vito jeng wrote:
>>> Matthias,
>>>
>>> Thank you for your assistance.
>>>
>>>> what is the status of this KIP?
>>>
>>> Unfortunately, there is no further progress.
>>> About seven weeks ago, I was injured in sports. I had a broken wrist on
>>> my left wrist.
>>> Many jobs are affected, including this KIP and implementation.
>>>
>>>
>>>> I just re-read it, and have a couple of follow up comments. Why do we
>>>> discuss the internal exceptions you want to add? Also, do we really need
>>>> them? Can't we just throw the correct exception directly instead of
>>>> wrapping it later?
>>>
>>> I think you may be right. As I say in the previous:
>>> "The original idea is that we can distinguish different state store
>>> exception for different handling. But to be honest, I am not quite sure
>>> this is necessary. Maybe have some change during implementation."
>>>
>>> During the implementation, I also feel we maybe not need wrapper it.
>>> We can just throw the correctly directly.
>>>
>>>
>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>
>>> When KafkaStream state is REBALANCING, I think it is a retryable error.
>>>
>>> StreamThreadStateStoreProvider#stores() will throw
>>> StreamThreadNotRunningException when StreamThread state is not RUNNING.
>> The
>>> user can retry until KafkaStream state is RUNNING.
>>>
>>>
>>>> When would we throw an `StateStoreEmptyException`? The semantics is
>>> unclear to me atm.
>>>
>>>> When the state is RUNNING, is `StateStoreClosedException` a retryable
>>> error?
>>>
>>> These two comments will be answered in another mail.
>>>
>>>
>>>
>>> ---
>>> Vito
>>>
>>> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Vito,
>>>>
>>>> what is the status of this KIP?
>>>>
>>>> I just re-read it, and have a couple of follow up comments. Why do we
>>>> discuss the internal exceptions you want to add? Also, do we really need
>>>> them? Can't we just throw the correct exception directly instead of
>>>> wrapping it later?
>>>>
>>>> When would we throw an `StateStoreEmptyException`? The semantics is
>>>> unclear to me atm.
>>>>
>>>> Is `StreamThreadNotRunningException` really an retryable error?
>>>>
>>>> When the state is RUNNING, is `StateStoreClosedException` a retryable
>>>> error?
>>>>
>>>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
>>>> time); that should be added
>>>>
>>>>
>>>> Overall I like the KIP but some details are still unclear. Maybe it
>>>> might help if you open an PR in parallel?
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 4/24/18 8:18 AM, vito jeng wrote:
>>>>> Hi, Guozhang,
>>>>>
>>>>> Thanks for the comment!
>>>>>
>>>>>
>>>>>
>>>>> Hi, Bill,
>>>>>
>>>>> I'll try to make some update to make the KIP better.
>>>>>
>>>>> Thanks for the comment!
>>>>>
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com>
>> wrote:
>>>>>
>>>>>> Hi Vito,
>>>>>>
>>>>>> Thanks for the KIP, overall it's a +1 from me.
>>>>>>
>>>>>> At this point, the only thing I would change is possibly removing the
>>>>>> listing of all methods called by the user and the listing of all store
>>>>>> types and focus on what states result in which exceptions thrown to
>> the
>>>>>> user.
>>>>>>
>>>>>> Thanks,
>>>>>> Bill
>>>>>>
>>>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> Thanks for the KIP Vito!
>>>>>>>
>>>>>>> I made a pass over the wiki and it looks great to me. I'm +1 on the
>>>> KIP.
>>>>>>>
>>>>>>> About the base class InvalidStateStoreException itself, I'd actually
>>>>>>> suggest we do not deprecate it but still expose it as part of the
>>>> public
>>>>>>> API, for people who do not want to handle these cases differently (if
>>>> we
>>>>>>> deprecate it then we are enforcing them to capture all three
>> exceptions
>>>>>>> one-by-one).
>>>>>>>
>>>>>>>
>>>>>>> Guozhang
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io>
>>>> wrote:
>>>>>>>
>>>>>>>> Hi Vito,
>>>>>>>>
>>>>>>>> Thanks for the KIP!
>>>>>>>>
>>>>>>>> I think it's much nicer to give callers different exceptions to tell
>>>>>> them
>>>>>>>> whether the state store got migrated, whether it's still
>> initializing,
>>>>>> or
>>>>>>>> whether there's some unrecoverable error.
>>>>>>>>
>>>>>>>> In the KIP, it's typically not necessary to discuss non-user-facing
>>>>>>> details
>>>>>>>> such as what exceptions we will throw internally. The KIP is
>> primarily
>>>>>> to
>>>>>>>> discuss public interface changes.
>>>>>>>>
>>>>>>>> You might consider simply removing all the internal details from the
>>>>>> KIP,
>>>>>>>> which will have the dual advantage that it makes the KIP smaller and
>>>>>>> easier
>>>>>>>> to agree on, as well as giving you more freedom in the internal
>>>> details
>>>>>>>> when it comes to implementation.
>>>>>>>>
>>>>>>>> I like your decision to have your refined exceptions extend
>>>>>>>> InvalidStateStoreException to ensure backward compatibility. Since
>> we
>>>>>>> want
>>>>>>>> to encourage callers to catch the more specific exceptions, and we
>>>>>> don't
>>>>>>>> expect to ever throw a raw InvalidStateStoreException anymore, you
>>>>>> might
>>>>>>>> consider adding the @Deprecated annotation to
>>>>>> InvalidStateStoreException.
>>>>>>>> This will gently encourage callers to migrate to the new exception
>> and
>>>>>>> open
>>>>>>>> the possibility of removing InvalidStateStoreException entirely in a
>>>>>>> future
>>>>>>>> release.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -John
>>>>>>>>
>>>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>>>>>> matthias@confluent.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for clarification! That makes sense to me.
>>>>>>>>>
>>>>>>>>> Can you update the KIP to make those suggestions explicit?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>>>>>> Matthias,
>>>>>>>>>>
>>>>>>>>>> Thanks for the feedback!
>>>>>>>>>>
>>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>>>
>>>>>>>>>> Got it!
>>>>>>>>>>
>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>> `StateStoreFailException`
>>>>>>> or
>>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
>>>>>>> suggest
>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
>>>>>> but
>>>>>>>> only
>>>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>>>
>>>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown at
>>>>>> all
>>>>>>>>>> anymore,
>>>>>>>>>> but only new sub-classes.
>>>>>>>>>>
>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>> The internal exception will be wrapped as category exception
>>>>>>> finally.
>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>
>>>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will throw
>>>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>>>>>> And then the internal exception will be wrapped as
>>>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException` during
>>>>>>> the
>>>>>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
>>>>>>> unclear
>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>
>>>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to distinguish
>>>>>>>> between
>>>>>>>>>> the different kinds of InvalidStateStoreException.
>>>>>>>>>> The original idea is that we can distinguish different state store
>>>>>>>>>> exception for
>>>>>>>>>> different handling.
>>>>>>>>>> But to be honest, I am not quite sure this is necessary. Maybe
>> have
>>>>>>>>>> some change during implementation.
>>>>>>>>>>
>>>>>>>>>> Does it make sense?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>>>>>> matthias@confluent.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the update Vito!
>>>>>>>>>>>
>>>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The (incomplete) question was, if we need
>>>>>> `StateStoreFailException`
>>>>>>> or
>>>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
>>>>>>> suggest
>>>>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
>>>>>> but
>>>>>>>> only
>>>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>>>
>>>>>>>>>>>> The internal exception will be wrapped as category exception
>>>>>>> finally.
>>>>>>>>>>>
>>>>>>>>>>> Can you elaborate?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
>>>>>>> unclear
>>>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the review.
>>>>>>>>>>>> I reply separately in the following sections.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Vito
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>>>>>> matthias@confluent.io
>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Seems you did a very thorough investigation of the code. It's
>>>>>>> useful
>>>>>>>>> to
>>>>>>>>>>>>> understand what user facing interfaces are affected.
>>>>>>>>>>>>
>>>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I also think too detailed. Especially the section `Changes in
>>>>>> call
>>>>>>>>>>> trace`.
>>>>>>>>>>>> Do you think it should be removed?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> To summarize my current understanding of your KIP, the main
>>>>>> change
>>>>>>>> is
>>>>>>>>> to
>>>>>>>>>>>>> introduce new exceptions that extend
>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
>>>>>>>>>>>> I think the best way is that all new exceptions extend from
>>>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Some questions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException` be
>>>>>> used
>>>>>>>> for
>>>>>>>>>>>>> this purpose?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Does this question miss some word?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
>>>>>>>> ngException`
>>>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>>>> ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> I apologize for not well written parts. I tried to modify some
>>>>>> code
>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>>> recent period and modify the KIP.
>>>>>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think you are right. We need to look at the state both of
>>>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> After further understanding of KafkaStreams thread and state
>>>>>>>> store,
>>>>>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not hosted
>>>>>> an
>>>>>>>> the
>>>>>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
>>>>>>> rebalance
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For the first case, users need to rediscover the store. For
>>>>>>> the
>>>>>>>>>>> second
>>>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
>>>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot rediscover or
>>>>>>>> retry.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user can
>>>>>> just
>>>>>>>>> plain
>>>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
>>>>>>> NOT_RUNNING,
>>>>>>>>> it
>>>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>>>> state store migrated to another instance, the user needs
>> to
>>>>>>>>>>>>> rediscover
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>>>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of the
>>>>>>> Jira
>>>>>>>>>>> was,
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
>>>>>>> strategies
>>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a user
>>>>>> just
>>>>>>>> need
>>>>>>>>>>> to
>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>>>> and can query the store later. On the other hand, if a
>>>>>> store
>>>>>>>> go
>>>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the store
>>>>>>>>> instead
>>>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks about
>>>>>>> this
>>>>>>>>>>> ideas
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
>>>>>> first
>>>>>>>> is
>>>>>>>>>>> what
>>>>>>>>>>>>>>>>>>>> categories of errors we want to notify the users, if we
>>>>>>> only
>>>>>>>>>>> wants
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
>>>>>>>> rename
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>>>>>> StateStoreClosedException
>>>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>>>> And then from there we should list what are the possible
>>>>>>>>> internal
>>>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call trace,
>>>>>> and
>>>>>>>>> which
>>>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and which
>>>>>> ones
>>>>>>>>>>> should
>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should not
>> be
>>>>>>>>> wrapped
>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>>>>>> different+errors
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -- Guozhang
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Guozhang Wang <wa...@gmail.com>.
I'm wondering if StreamThreadNotStarted could be merged into
StreamThreadNotRunning, because I think users' handling logic for the third
case would be likely the same as the second. Do you have some scenarios
where users may want to handle them differently?
Guozhang
On Sun, Jun 24, 2018 at 5:25 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Sorry to hear! Get well soon!
>
> It's not a big deal if the KIP stalls a little bit. Feel free to pick it
> up again when you find time.
>
> >>> Is `StreamThreadNotRunningException` really an retryable error?
> >>
> >> When KafkaStream state is REBALANCING, I think it is a retryable error.
> >>
> >> StreamThreadStateStoreProvider#stores() will throw
> >> StreamThreadNotRunningException when StreamThread state is not
> RUNNING. The
> >> user can retry until KafkaStream state is RUNNING.
>
> I see. If this is the intention, than I would suggest to have two (or
> maybe three) different exceptions:
>
> - StreamThreadRebalancingException (retryable)
> - StreamThreadNotRunning (not retryable -- thrown if in state
> PENDING_SHUTDOWN or DEAD
> - maybe StreamThreadNotStarted (for state CREATED)
>
> The last one is tricky and could also be merged into one of the first
> two, depending if you want to argue that it's retryable or not. (Just
> food for though -- not sure what others think.)
>
>
>
> -Matthias
>
> On 6/22/18 8:06 AM, vito jeng wrote:
> > Matthias,
> >
> > Thank you for your assistance.
> >
> >> what is the status of this KIP?
> >
> > Unfortunately, there is no further progress.
> > About seven weeks ago, I was injured in sports. I had a broken wrist on
> > my left wrist.
> > Many jobs are affected, including this KIP and implementation.
> >
> >
> >> I just re-read it, and have a couple of follow up comments. Why do we
> >> discuss the internal exceptions you want to add? Also, do we really need
> >> them? Can't we just throw the correct exception directly instead of
> >> wrapping it later?
> >
> > I think you may be right. As I say in the previous:
> > "The original idea is that we can distinguish different state store
> > exception for different handling. But to be honest, I am not quite sure
> > this is necessary. Maybe have some change during implementation."
> >
> > During the implementation, I also feel we maybe not need wrapper it.
> > We can just throw the correctly directly.
> >
> >
> >> Is `StreamThreadNotRunningException` really an retryable error?
> >
> > When KafkaStream state is REBALANCING, I think it is a retryable error.
> >
> > StreamThreadStateStoreProvider#stores() will throw
> > StreamThreadNotRunningException when StreamThread state is not RUNNING.
> The
> > user can retry until KafkaStream state is RUNNING.
> >
> >
> >> When would we throw an `StateStoreEmptyException`? The semantics is
> > unclear to me atm.
> >
> >> When the state is RUNNING, is `StateStoreClosedException` a retryable
> > error?
> >
> > These two comments will be answered in another mail.
> >
> >
> >
> > ---
> > Vito
> >
> > On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Vito,
> >>
> >> what is the status of this KIP?
> >>
> >> I just re-read it, and have a couple of follow up comments. Why do we
> >> discuss the internal exceptions you want to add? Also, do we really need
> >> them? Can't we just throw the correct exception directly instead of
> >> wrapping it later?
> >>
> >> When would we throw an `StateStoreEmptyException`? The semantics is
> >> unclear to me atm.
> >>
> >> Is `StreamThreadNotRunningException` really an retryable error?
> >>
> >> When the state is RUNNING, is `StateStoreClosedException` a retryable
> >> error?
> >>
> >> One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
> >> time); that should be added
> >>
> >>
> >> Overall I like the KIP but some details are still unclear. Maybe it
> >> might help if you open an PR in parallel?
> >>
> >>
> >> -Matthias
> >>
> >> On 4/24/18 8:18 AM, vito jeng wrote:
> >>> Hi, Guozhang,
> >>>
> >>> Thanks for the comment!
> >>>
> >>>
> >>>
> >>> Hi, Bill,
> >>>
> >>> I'll try to make some update to make the KIP better.
> >>>
> >>> Thanks for the comment!
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com>
> wrote:
> >>>
> >>>> Hi Vito,
> >>>>
> >>>> Thanks for the KIP, overall it's a +1 from me.
> >>>>
> >>>> At this point, the only thing I would change is possibly removing the
> >>>> listing of all methods called by the user and the listing of all store
> >>>> types and focus on what states result in which exceptions thrown to
> the
> >>>> user.
> >>>>
> >>>> Thanks,
> >>>> Bill
> >>>>
> >>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >>>>
> >>>>> Thanks for the KIP Vito!
> >>>>>
> >>>>> I made a pass over the wiki and it looks great to me. I'm +1 on the
> >> KIP.
> >>>>>
> >>>>> About the base class InvalidStateStoreException itself, I'd actually
> >>>>> suggest we do not deprecate it but still expose it as part of the
> >> public
> >>>>> API, for people who do not want to handle these cases differently (if
> >> we
> >>>>> deprecate it then we are enforcing them to capture all three
> exceptions
> >>>>> one-by-one).
> >>>>>
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io>
> >> wrote:
> >>>>>
> >>>>>> Hi Vito,
> >>>>>>
> >>>>>> Thanks for the KIP!
> >>>>>>
> >>>>>> I think it's much nicer to give callers different exceptions to tell
> >>>> them
> >>>>>> whether the state store got migrated, whether it's still
> initializing,
> >>>> or
> >>>>>> whether there's some unrecoverable error.
> >>>>>>
> >>>>>> In the KIP, it's typically not necessary to discuss non-user-facing
> >>>>> details
> >>>>>> such as what exceptions we will throw internally. The KIP is
> primarily
> >>>> to
> >>>>>> discuss public interface changes.
> >>>>>>
> >>>>>> You might consider simply removing all the internal details from the
> >>>> KIP,
> >>>>>> which will have the dual advantage that it makes the KIP smaller and
> >>>>> easier
> >>>>>> to agree on, as well as giving you more freedom in the internal
> >> details
> >>>>>> when it comes to implementation.
> >>>>>>
> >>>>>> I like your decision to have your refined exceptions extend
> >>>>>> InvalidStateStoreException to ensure backward compatibility. Since
> we
> >>>>> want
> >>>>>> to encourage callers to catch the more specific exceptions, and we
> >>>> don't
> >>>>>> expect to ever throw a raw InvalidStateStoreException anymore, you
> >>>> might
> >>>>>> consider adding the @Deprecated annotation to
> >>>> InvalidStateStoreException.
> >>>>>> This will gently encourage callers to migrate to the new exception
> and
> >>>>> open
> >>>>>> the possibility of removing InvalidStateStoreException entirely in a
> >>>>> future
> >>>>>> release.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> -John
> >>>>>>
> >>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >>>> matthias@confluent.io>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for clarification! That makes sense to me.
> >>>>>>>
> >>>>>>> Can you update the KIP to make those suggestions explicit?
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>>>> Matthias,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback!
> >>>>>>>>
> >>>>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>>>
> >>>>>>>> Got it!
> >>>>>>>>
> >>>>>>>>> The (incomplete) question was, if we need
> >>>> `StateStoreFailException`
> >>>>> or
> >>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>>>> suggest
> >>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >>>> but
> >>>>>> only
> >>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>
> >>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown at
> >>>> all
> >>>>>>>> anymore,
> >>>>>>>> but only new sub-classes.
> >>>>>>>>
> >>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>> The internal exception will be wrapped as category exception
> >>>>> finally.
> >>>>>>>>> Can you elaborate?
> >>>>>>>>
> >>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will throw
> >>>>>>>> `StreamThreadNotRunningException`(internal exception).
> >>>>>>>> And then the internal exception will be wrapped as
> >>>>>>>> `StateStoreRetryableException` or `StateStoreFailException` during
> >>>>> the
> >>>>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>>>> unclear
> >>>>>>>> to me atm why they are introduced.
> >>>>>>>>
> >>>>>>>> Hmmm...the purpose of the "internal exceptions" is to distinguish
> >>>>>> between
> >>>>>>>> the different kinds of InvalidStateStoreException.
> >>>>>>>> The original idea is that we can distinguish different state store
> >>>>>>>> exception for
> >>>>>>>> different handling.
> >>>>>>>> But to be honest, I am not quite sure this is necessary. Maybe
> have
> >>>>>>>> some change during implementation.
> >>>>>>>>
> >>>>>>>> Does it make sense?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Vito
> >>>>>>>>
> >>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>>>> matthias@confluent.io>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks for the update Vito!
> >>>>>>>>>
> >>>>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The (incomplete) question was, if we need
> >>>> `StateStoreFailException`
> >>>>> or
> >>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>>>> suggest
> >>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >>>> but
> >>>>>> only
> >>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>
> >>>>>>>>>> The internal exception will be wrapped as category exception
> >>>>> finally.
> >>>>>>>>>
> >>>>>>>>> Can you elaborate?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>>>> unclear
> >>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>>>> Matthias,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the review.
> >>>>>>>>>> I reply separately in the following sections.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> Vito
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>>>> matthias@confluent.io
> >>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
> >>>>>>>>>>>
> >>>>>>>>>>> Seems you did a very thorough investigation of the code. It's
> >>>>> useful
> >>>>>>> to
> >>>>>>>>>>> understand what user facing interfaces are affected.
> >>>>>>>>>>
> >>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I also think too detailed. Especially the section `Changes in
> >>>> call
> >>>>>>>>> trace`.
> >>>>>>>>>> Do you think it should be removed?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> To summarize my current understanding of your KIP, the main
> >>>> change
> >>>>>> is
> >>>>>>> to
> >>>>>>>>>>> introduce new exceptions that extend
> >>>> `InvalidStateStoreException`.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> yep. Keep compatibility in this KIP is important things.
> >>>>>>>>>> I think the best way is that all new exceptions extend from
> >>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Some questions:
> >>>>>>>>>>>
> >>>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException` be
> >>>> used
> >>>>>> for
> >>>>>>>>>>> this purpose?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Does this question miss some word?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
> >>>>>> ngException`
> >>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>> ?
> >>>>>>>>>>>
> >>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> I apologize for not well written parts. I tried to modify some
> >>>> code
> >>>>>> in
> >>>>>>>>> the
> >>>>>>>>>> recent period and modify the KIP.
> >>>>>>>>>> The modification is now complete. Please look again.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -Matthias
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think you are right. We need to look at the state both of
> >>>>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> After further understanding of KafkaStreams thread and state
> >>>>>> store,
> >>>>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - either a store got migrated way and thus, is not hosted
> >>>> an
> >>>>>> the
> >>>>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
> >>>>> rebalance
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For the first case, users need to rediscover the store. For
> >>>>> the
> >>>>>>>>> second
> >>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> >>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>> uses cannot query at all and thus they cannot rediscover or
> >>>>>> retry.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user can
> >>>> just
> >>>>>>> plain
> >>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> >>>>> NOT_RUNNING,
> >>>>>>> it
> >>>>>>>>>>>>> means
> >>>>>>>>>>>>>>>> state store migrated to another instance, the user needs
> to
> >>>>>>>>>>> rediscover
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of the
> >>>>> Jira
> >>>>>>>>> was,
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
> >>>>> strategies
> >>>>>> to
> >>>>>>>>> the
> >>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a user
> >>>> just
> >>>>>> need
> >>>>>>>>> to
> >>>>>>>>>>>>> wait
> >>>>>>>>>>>>>>>>> and can query the store later. On the other hand, if a
> >>>> store
> >>>>>> go
> >>>>>>>>>>>>> migrated
> >>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the store
> >>>>>>> instead
> >>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks about
> >>>>> this
> >>>>>>>>> ideas
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
> >>>> first
> >>>>>> is
> >>>>>>>>> what
> >>>>>>>>>>>>>>>>>> categories of errors we want to notify the users, if we
> >>>>> only
> >>>>>>>>> wants
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
> >>>>>> rename
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>>>> StateStoreClosedException
> >>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>> And then from there we should list what are the possible
> >>>>>>> internal
> >>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call trace,
> >>>> and
> >>>>>>> which
> >>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and which
> >>>> ones
> >>>>>>>>> should
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should not
> be
> >>>>>>> wrapped
> >>>>>>>>>>> at
> >>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> >>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>>>> different+errors
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
--
-- Guozhang
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
> Sorry to hear! Get well soon!
I'm back! :)
Base on many suggestion and discussion, I already rewrite the KIP.
Please take a look:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors
> I see. If this is the intention, than I would suggest to have two (or
> maybe three) different exceptions:
> - StreamThreadRebalancingException (retryable)
> - StreamThreadNotRunning (not retryable -- thrown if in state
> PENDING_SHUTDOWN or DEAD
> - maybe StreamThreadNotStarted (for state CREATED)
> The last one is tricky and could also be merged into one of the first
> two, depending if you want to argue that it's retryable or not. (Just
> food for though -- not sure what others think.)
I agree with your suggestion, really helpful.
> Overall I like the KIP but some details are still unclear. Maybe it
> might help if you open an PR in parallel?
It would be a lot of help for further discussion.
I will complete the implementation in a few days and then I will create a
PR.
---
Vito
On Mon, Jun 25, 2018 at 8:25 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Sorry to hear! Get well soon!
>
> It's not a big deal if the KIP stalls a little bit. Feel free to pick it
> up again when you find time.
>
> >>> Is `StreamThreadNotRunningException` really an retryable error?
> >>
> >> When KafkaStream state is REBALANCING, I think it is a retryable error.
> >>
> >> StreamThreadStateStoreProvider#stores() will throw
> >> StreamThreadNotRunningException when StreamThread state is not RUNNING.
> The
> >> user can retry until KafkaStream state is RUNNING.
>
> I see. If this is the intention, than I would suggest to have two (or
> maybe three) different exceptions:
>
> - StreamThreadRebalancingException (retryable)
> - StreamThreadNotRunning (not retryable -- thrown if in state
> PENDING_SHUTDOWN or DEAD
> - maybe StreamThreadNotStarted (for state CREATED)
>
> The last one is tricky and could also be merged into one of the first
> two, depending if you want to argue that it's retryable or not. (Just
> food for though -- not sure what others think.)
>
>
>
> -Matthias
>
> On 6/22/18 8:06 AM, vito jeng wrote:
> > Matthias,
> >
> > Thank you for your assistance.
> >
> >> what is the status of this KIP?
> >
> > Unfortunately, there is no further progress.
> > About seven weeks ago, I was injured in sports. I had a broken wrist on
> > my left wrist.
> > Many jobs are affected, including this KIP and implementation.
> >
> >
> >> I just re-read it, and have a couple of follow up comments. Why do we
> >> discuss the internal exceptions you want to add? Also, do we really need
> >> them? Can't we just throw the correct exception directly instead of
> >> wrapping it later?
> >
> > I think you may be right. As I say in the previous:
> > "The original idea is that we can distinguish different state store
> > exception for different handling. But to be honest, I am not quite sure
> > this is necessary. Maybe have some change during implementation."
> >
> > During the implementation, I also feel we maybe not need wrapper it.
> > We can just throw the correctly directly.
> >
> >
> >> Is `StreamThreadNotRunningException` really an retryable error?
> >
> > When KafkaStream state is REBALANCING, I think it is a retryable error.
> >
> > StreamThreadStateStoreProvider#stores() will throw
> > StreamThreadNotRunningException when StreamThread state is not RUNNING.
> The
> > user can retry until KafkaStream state is RUNNING.
> >
> >
> >> When would we throw an `StateStoreEmptyException`? The semantics is
> > unclear to me atm.
> >
> >> When the state is RUNNING, is `StateStoreClosedException` a retryable
> > error?
> >
> > These two comments will be answered in another mail.
> >
> >
> >
> > ---
> > Vito
> >
> > On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Vito,
> >>
> >> what is the status of this KIP?
> >>
> >> I just re-read it, and have a couple of follow up comments. Why do we
> >> discuss the internal exceptions you want to add? Also, do we really need
> >> them? Can't we just throw the correct exception directly instead of
> >> wrapping it later?
> >>
> >> When would we throw an `StateStoreEmptyException`? The semantics is
> >> unclear to me atm.
> >>
> >> Is `StreamThreadNotRunningException` really an retryable error?
> >>
> >> When the state is RUNNING, is `StateStoreClosedException` a retryable
> >> error?
> >>
> >> One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
> >> time); that should be added
> >>
> >>
> >> Overall I like the KIP but some details are still unclear. Maybe it
> >> might help if you open an PR in parallel?
> >>
> >>
> >> -Matthias
> >>
> >> On 4/24/18 8:18 AM, vito jeng wrote:
> >>> Hi, Guozhang,
> >>>
> >>> Thanks for the comment!
> >>>
> >>>
> >>>
> >>> Hi, Bill,
> >>>
> >>> I'll try to make some update to make the KIP better.
> >>>
> >>> Thanks for the comment!
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com>
> wrote:
> >>>
> >>>> Hi Vito,
> >>>>
> >>>> Thanks for the KIP, overall it's a +1 from me.
> >>>>
> >>>> At this point, the only thing I would change is possibly removing the
> >>>> listing of all methods called by the user and the listing of all store
> >>>> types and focus on what states result in which exceptions thrown to
> the
> >>>> user.
> >>>>
> >>>> Thanks,
> >>>> Bill
> >>>>
> >>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >>>>
> >>>>> Thanks for the KIP Vito!
> >>>>>
> >>>>> I made a pass over the wiki and it looks great to me. I'm +1 on the
> >> KIP.
> >>>>>
> >>>>> About the base class InvalidStateStoreException itself, I'd actually
> >>>>> suggest we do not deprecate it but still expose it as part of the
> >> public
> >>>>> API, for people who do not want to handle these cases differently (if
> >> we
> >>>>> deprecate it then we are enforcing them to capture all three
> exceptions
> >>>>> one-by-one).
> >>>>>
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io>
> >> wrote:
> >>>>>
> >>>>>> Hi Vito,
> >>>>>>
> >>>>>> Thanks for the KIP!
> >>>>>>
> >>>>>> I think it's much nicer to give callers different exceptions to tell
> >>>> them
> >>>>>> whether the state store got migrated, whether it's still
> initializing,
> >>>> or
> >>>>>> whether there's some unrecoverable error.
> >>>>>>
> >>>>>> In the KIP, it's typically not necessary to discuss non-user-facing
> >>>>> details
> >>>>>> such as what exceptions we will throw internally. The KIP is
> primarily
> >>>> to
> >>>>>> discuss public interface changes.
> >>>>>>
> >>>>>> You might consider simply removing all the internal details from the
> >>>> KIP,
> >>>>>> which will have the dual advantage that it makes the KIP smaller and
> >>>>> easier
> >>>>>> to agree on, as well as giving you more freedom in the internal
> >> details
> >>>>>> when it comes to implementation.
> >>>>>>
> >>>>>> I like your decision to have your refined exceptions extend
> >>>>>> InvalidStateStoreException to ensure backward compatibility. Since
> we
> >>>>> want
> >>>>>> to encourage callers to catch the more specific exceptions, and we
> >>>> don't
> >>>>>> expect to ever throw a raw InvalidStateStoreException anymore, you
> >>>> might
> >>>>>> consider adding the @Deprecated annotation to
> >>>> InvalidStateStoreException.
> >>>>>> This will gently encourage callers to migrate to the new exception
> and
> >>>>> open
> >>>>>> the possibility of removing InvalidStateStoreException entirely in a
> >>>>> future
> >>>>>> release.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> -John
> >>>>>>
> >>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >>>> matthias@confluent.io>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for clarification! That makes sense to me.
> >>>>>>>
> >>>>>>> Can you update the KIP to make those suggestions explicit?
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>>>> Matthias,
> >>>>>>>>
> >>>>>>>> Thanks for the feedback!
> >>>>>>>>
> >>>>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>>>
> >>>>>>>> Got it!
> >>>>>>>>
> >>>>>>>>> The (incomplete) question was, if we need
> >>>> `StateStoreFailException`
> >>>>> or
> >>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>>>> suggest
> >>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >>>> but
> >>>>>> only
> >>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>
> >>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown at
> >>>> all
> >>>>>>>> anymore,
> >>>>>>>> but only new sub-classes.
> >>>>>>>>
> >>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>> The internal exception will be wrapped as category exception
> >>>>> finally.
> >>>>>>>>> Can you elaborate?
> >>>>>>>>
> >>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will throw
> >>>>>>>> `StreamThreadNotRunningException`(internal exception).
> >>>>>>>> And then the internal exception will be wrapped as
> >>>>>>>> `StateStoreRetryableException` or `StateStoreFailException` during
> >>>>> the
> >>>>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>>>> unclear
> >>>>>>>> to me atm why they are introduced.
> >>>>>>>>
> >>>>>>>> Hmmm...the purpose of the "internal exceptions" is to distinguish
> >>>>>> between
> >>>>>>>> the different kinds of InvalidStateStoreException.
> >>>>>>>> The original idea is that we can distinguish different state store
> >>>>>>>> exception for
> >>>>>>>> different handling.
> >>>>>>>> But to be honest, I am not quite sure this is necessary. Maybe
> have
> >>>>>>>> some change during implementation.
> >>>>>>>>
> >>>>>>>> Does it make sense?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Vito
> >>>>>>>>
> >>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>>>> matthias@confluent.io>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks for the update Vito!
> >>>>>>>>>
> >>>>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The (incomplete) question was, if we need
> >>>> `StateStoreFailException`
> >>>>> or
> >>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>>>> suggest
> >>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >>>> but
> >>>>>> only
> >>>>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Not sure what this sentence means:
> >>>>>>>>>
> >>>>>>>>>> The internal exception will be wrapped as category exception
> >>>>> finally.
> >>>>>>>>>
> >>>>>>>>> Can you elaborate?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>>>> unclear
> >>>>>>>>> to me atm why they are introduced.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>>>> Matthias,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the review.
> >>>>>>>>>> I reply separately in the following sections.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> Vito
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>>>> matthias@confluent.io
> >>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
> >>>>>>>>>>>
> >>>>>>>>>>> Seems you did a very thorough investigation of the code. It's
> >>>>> useful
> >>>>>>> to
> >>>>>>>>>>> understand what user facing interfaces are affected.
> >>>>>>>>>>
> >>>>>>>>>> (Some parts might be even too detailed for a KIP.)
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I also think too detailed. Especially the section `Changes in
> >>>> call
> >>>>>>>>> trace`.
> >>>>>>>>>> Do you think it should be removed?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> To summarize my current understanding of your KIP, the main
> >>>> change
> >>>>>> is
> >>>>>>> to
> >>>>>>>>>>> introduce new exceptions that extend
> >>>> `InvalidStateStoreException`.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> yep. Keep compatibility in this KIP is important things.
> >>>>>>>>>> I think the best way is that all new exceptions extend from
> >>>>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Some questions:
> >>>>>>>>>>>
> >>>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException` be
> >>>> used
> >>>>>> for
> >>>>>>>>>>> this purpose?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Does this question miss some word?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
> >>>>>> ngException`
> >>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>>>> `StateStoreFailException`
> >>>>>>>>>>> ?
> >>>>>>>>>>>
> >>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> I apologize for not well written parts. I tried to modify some
> >>>> code
> >>>>>> in
> >>>>>>>>> the
> >>>>>>>>>> recent period and modify the KIP.
> >>>>>>>>>> The modification is now complete. Please look again.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -Matthias
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I think you are right. We need to look at the state both of
> >>>>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> After further understanding of KafkaStreams thread and state
> >>>>>> store,
> >>>>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - either a store got migrated way and thus, is not hosted
> >>>> an
> >>>>>> the
> >>>>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
> >>>>> rebalance
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For the first case, users need to rediscover the store. For
> >>>>> the
> >>>>>>>>> second
> >>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> >>>>>>> NOT_RUNNING,
> >>>>>>>>>>>>>>> uses cannot query at all and thus they cannot rediscover or
> >>>>>> retry.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user can
> >>>> just
> >>>>>>> plain
> >>>>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> >>>>> NOT_RUNNING,
> >>>>>>> it
> >>>>>>>>>>>>> means
> >>>>>>>>>>>>>>>> state store migrated to another instance, the user needs
> to
> >>>>>>>>>>> rediscover
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of the
> >>>>> Jira
> >>>>>>>>> was,
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
> >>>>> strategies
> >>>>>> to
> >>>>>>>>> the
> >>>>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a user
> >>>> just
> >>>>>> need
> >>>>>>>>> to
> >>>>>>>>>>>>> wait
> >>>>>>>>>>>>>>>>> and can query the store later. On the other hand, if a
> >>>> store
> >>>>>> go
> >>>>>>>>>>>>> migrated
> >>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the store
> >>>>>>> instead
> >>>>>>>>>>> of a
> >>>>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks about
> >>>>> this
> >>>>>>>>> ideas
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
> >>>> first
> >>>>>> is
> >>>>>>>>> what
> >>>>>>>>>>>>>>>>>> categories of errors we want to notify the users, if we
> >>>>> only
> >>>>>>>>> wants
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
> >>>>>> rename
> >>>>>>>>> the
> >>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>>>> StateStoreClosedException
> >>>>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>>>> And then from there we should list what are the possible
> >>>>>>> internal
> >>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call trace,
> >>>> and
> >>>>>>> which
> >>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and which
> >>>> ones
> >>>>>>>>> should
> >>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should not
> be
> >>>>>>> wrapped
> >>>>>>>>>>> at
> >>>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> >>>>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>>>> different+errors
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -- Guozhang
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Sorry to hear! Get well soon!
It's not a big deal if the KIP stalls a little bit. Feel free to pick it
up again when you find time.
>>> Is `StreamThreadNotRunningException` really an retryable error?
>>
>> When KafkaStream state is REBALANCING, I think it is a retryable error.
>>
>> StreamThreadStateStoreProvider#stores() will throw
>> StreamThreadNotRunningException when StreamThread state is not RUNNING. The
>> user can retry until KafkaStream state is RUNNING.
I see. If this is the intention, than I would suggest to have two (or
maybe three) different exceptions:
- StreamThreadRebalancingException (retryable)
- StreamThreadNotRunning (not retryable -- thrown if in state
PENDING_SHUTDOWN or DEAD
- maybe StreamThreadNotStarted (for state CREATED)
The last one is tricky and could also be merged into one of the first
two, depending if you want to argue that it's retryable or not. (Just
food for though -- not sure what others think.)
-Matthias
On 6/22/18 8:06 AM, vito jeng wrote:
> Matthias,
>
> Thank you for your assistance.
>
>> what is the status of this KIP?
>
> Unfortunately, there is no further progress.
> About seven weeks ago, I was injured in sports. I had a broken wrist on
> my left wrist.
> Many jobs are affected, including this KIP and implementation.
>
>
>> I just re-read it, and have a couple of follow up comments. Why do we
>> discuss the internal exceptions you want to add? Also, do we really need
>> them? Can't we just throw the correct exception directly instead of
>> wrapping it later?
>
> I think you may be right. As I say in the previous:
> "The original idea is that we can distinguish different state store
> exception for different handling. But to be honest, I am not quite sure
> this is necessary. Maybe have some change during implementation."
>
> During the implementation, I also feel we maybe not need wrapper it.
> We can just throw the correctly directly.
>
>
>> Is `StreamThreadNotRunningException` really an retryable error?
>
> When KafkaStream state is REBALANCING, I think it is a retryable error.
>
> StreamThreadStateStoreProvider#stores() will throw
> StreamThreadNotRunningException when StreamThread state is not RUNNING. The
> user can retry until KafkaStream state is RUNNING.
>
>
>> When would we throw an `StateStoreEmptyException`? The semantics is
> unclear to me atm.
>
>> When the state is RUNNING, is `StateStoreClosedException` a retryable
> error?
>
> These two comments will be answered in another mail.
>
>
>
> ---
> Vito
>
> On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Vito,
>>
>> what is the status of this KIP?
>>
>> I just re-read it, and have a couple of follow up comments. Why do we
>> discuss the internal exceptions you want to add? Also, do we really need
>> them? Can't we just throw the correct exception directly instead of
>> wrapping it later?
>>
>> When would we throw an `StateStoreEmptyException`? The semantics is
>> unclear to me atm.
>>
>> Is `StreamThreadNotRunningException` really an retryable error?
>>
>> When the state is RUNNING, is `StateStoreClosedException` a retryable
>> error?
>>
>> One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
>> time); that should be added
>>
>>
>> Overall I like the KIP but some details are still unclear. Maybe it
>> might help if you open an PR in parallel?
>>
>>
>> -Matthias
>>
>> On 4/24/18 8:18 AM, vito jeng wrote:
>>> Hi, Guozhang,
>>>
>>> Thanks for the comment!
>>>
>>>
>>>
>>> Hi, Bill,
>>>
>>> I'll try to make some update to make the KIP better.
>>>
>>> Thanks for the comment!
>>>
>>>
>>> ---
>>> Vito
>>>
>>> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com> wrote:
>>>
>>>> Hi Vito,
>>>>
>>>> Thanks for the KIP, overall it's a +1 from me.
>>>>
>>>> At this point, the only thing I would change is possibly removing the
>>>> listing of all methods called by the user and the listing of all store
>>>> types and focus on what states result in which exceptions thrown to the
>>>> user.
>>>>
>>>> Thanks,
>>>> Bill
>>>>
>>>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>>>>
>>>>> Thanks for the KIP Vito!
>>>>>
>>>>> I made a pass over the wiki and it looks great to me. I'm +1 on the
>> KIP.
>>>>>
>>>>> About the base class InvalidStateStoreException itself, I'd actually
>>>>> suggest we do not deprecate it but still expose it as part of the
>> public
>>>>> API, for people who do not want to handle these cases differently (if
>> we
>>>>> deprecate it then we are enforcing them to capture all three exceptions
>>>>> one-by-one).
>>>>>
>>>>>
>>>>> Guozhang
>>>>>
>>>>>
>>>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io>
>> wrote:
>>>>>
>>>>>> Hi Vito,
>>>>>>
>>>>>> Thanks for the KIP!
>>>>>>
>>>>>> I think it's much nicer to give callers different exceptions to tell
>>>> them
>>>>>> whether the state store got migrated, whether it's still initializing,
>>>> or
>>>>>> whether there's some unrecoverable error.
>>>>>>
>>>>>> In the KIP, it's typically not necessary to discuss non-user-facing
>>>>> details
>>>>>> such as what exceptions we will throw internally. The KIP is primarily
>>>> to
>>>>>> discuss public interface changes.
>>>>>>
>>>>>> You might consider simply removing all the internal details from the
>>>> KIP,
>>>>>> which will have the dual advantage that it makes the KIP smaller and
>>>>> easier
>>>>>> to agree on, as well as giving you more freedom in the internal
>> details
>>>>>> when it comes to implementation.
>>>>>>
>>>>>> I like your decision to have your refined exceptions extend
>>>>>> InvalidStateStoreException to ensure backward compatibility. Since we
>>>>> want
>>>>>> to encourage callers to catch the more specific exceptions, and we
>>>> don't
>>>>>> expect to ever throw a raw InvalidStateStoreException anymore, you
>>>> might
>>>>>> consider adding the @Deprecated annotation to
>>>> InvalidStateStoreException.
>>>>>> This will gently encourage callers to migrate to the new exception and
>>>>> open
>>>>>> the possibility of removing InvalidStateStoreException entirely in a
>>>>> future
>>>>>> release.
>>>>>>
>>>>>> Thanks,
>>>>>> -John
>>>>>>
>>>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>>>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for clarification! That makes sense to me.
>>>>>>>
>>>>>>> Can you update the KIP to make those suggestions explicit?
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>>>> Matthias,
>>>>>>>>
>>>>>>>> Thanks for the feedback!
>>>>>>>>
>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>
>>>>>>>> Got it!
>>>>>>>>
>>>>>>>>> The (incomplete) question was, if we need
>>>> `StateStoreFailException`
>>>>> or
>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
>>>>> suggest
>>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
>>>> but
>>>>>> only
>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>
>>>>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown at
>>>> all
>>>>>>>> anymore,
>>>>>>>> but only new sub-classes.
>>>>>>>>
>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>> The internal exception will be wrapped as category exception
>>>>> finally.
>>>>>>>>> Can you elaborate?
>>>>>>>>
>>>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will throw
>>>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>>>> And then the internal exception will be wrapped as
>>>>>>>> `StateStoreRetryableException` or `StateStoreFailException` during
>>>>> the
>>>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
>>>>> unclear
>>>>>>>> to me atm why they are introduced.
>>>>>>>>
>>>>>>>> Hmmm...the purpose of the "internal exceptions" is to distinguish
>>>>>> between
>>>>>>>> the different kinds of InvalidStateStoreException.
>>>>>>>> The original idea is that we can distinguish different state store
>>>>>>>> exception for
>>>>>>>> different handling.
>>>>>>>> But to be honest, I am not quite sure this is necessary. Maybe have
>>>>>>>> some change during implementation.
>>>>>>>>
>>>>>>>> Does it make sense?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>>>> matthias@confluent.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the update Vito!
>>>>>>>>>
>>>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The (incomplete) question was, if we need
>>>> `StateStoreFailException`
>>>>> or
>>>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
>>>>> suggest
>>>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
>>>> but
>>>>>> only
>>>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Not sure what this sentence means:
>>>>>>>>>
>>>>>>>>>> The internal exception will be wrapped as category exception
>>>>> finally.
>>>>>>>>>
>>>>>>>>> Can you elaborate?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you explain the purpose of the "internal exceptions". It's
>>>>> unclear
>>>>>>>>> to me atm why they are introduced.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>>>> Matthias,
>>>>>>>>>>
>>>>>>>>>> Thanks for the review.
>>>>>>>>>> I reply separately in the following sections.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>>>> matthias@confluent.io
>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
>>>>>>>>>>>
>>>>>>>>>>> Seems you did a very thorough investigation of the code. It's
>>>>> useful
>>>>>>> to
>>>>>>>>>>> understand what user facing interfaces are affected.
>>>>>>>>>>
>>>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I also think too detailed. Especially the section `Changes in
>>>> call
>>>>>>>>> trace`.
>>>>>>>>>> Do you think it should be removed?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> To summarize my current understanding of your KIP, the main
>>>> change
>>>>>> is
>>>>>>> to
>>>>>>>>>>> introduce new exceptions that extend
>>>> `InvalidStateStoreException`.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> yep. Keep compatibility in this KIP is important things.
>>>>>>>>>> I think the best way is that all new exceptions extend from
>>>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Some questions:
>>>>>>>>>>>
>>>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException` be
>>>> used
>>>>>> for
>>>>>>>>>>> this purpose?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Does this question miss some word?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
>>>>>> ngException`
>>>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>>>> `StateStoreFailException`
>>>>>>>>>>> ?
>>>>>>>>>>>
>>>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I apologize for not well written parts. I tried to modify some
>>>> code
>>>>>> in
>>>>>>>>> the
>>>>>>>>>> recent period and modify the KIP.
>>>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Vito
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think you are right. We need to look at the state both of
>>>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> After further understanding of KafkaStreams thread and state
>>>>>> store,
>>>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - either a store got migrated way and thus, is not hosted
>>>> an
>>>>>> the
>>>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
>>>>> rebalance
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For the first case, users need to rediscover the store. For
>>>>> the
>>>>>>>>> second
>>>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
>>>>>>> NOT_RUNNING,
>>>>>>>>>>>>>>> uses cannot query at all and thus they cannot rediscover or
>>>>>> retry.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user can
>>>> just
>>>>>>> plain
>>>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
>>>>> NOT_RUNNING,
>>>>>>> it
>>>>>>>>>>>>> means
>>>>>>>>>>>>>>>> state store migrated to another instance, the user needs to
>>>>>>>>>>> rediscover
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>>>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of the
>>>>> Jira
>>>>>>>>> was,
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> give different exceptions for different "recovery"
>>>>> strategies
>>>>>> to
>>>>>>>>> the
>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For example, if a store is currently recreated, a user
>>>> just
>>>>>> need
>>>>>>>>> to
>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>> and can query the store later. On the other hand, if a
>>>> store
>>>>>> go
>>>>>>>>>>>>> migrated
>>>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the store
>>>>>>> instead
>>>>>>>>>>> of a
>>>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks about
>>>>> this
>>>>>>>>> ideas
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
>>>> first
>>>>>> is
>>>>>>>>> what
>>>>>>>>>>>>>>>>>> categories of errors we want to notify the users, if we
>>>>> only
>>>>>>>>> wants
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
>>>>>> rename
>>>>>>>>> the
>>>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>>>> StateStoreClosedException
>>>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>>>> And then from there we should list what are the possible
>>>>>>> internal
>>>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call trace,
>>>> and
>>>>>>> which
>>>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and which
>>>> ones
>>>>>>>>> should
>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should not be
>>>>>>> wrapped
>>>>>>>>>>> at
>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>>>> different+errors
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -- Guozhang
>>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
Thank you for your assistance.
> what is the status of this KIP?
Unfortunately, there is no further progress.
About seven weeks ago, I was injured in sports. I had a broken wrist on
my left wrist.
Many jobs are affected, including this KIP and implementation.
> I just re-read it, and have a couple of follow up comments. Why do we
> discuss the internal exceptions you want to add? Also, do we really need
> them? Can't we just throw the correct exception directly instead of
> wrapping it later?
I think you may be right. As I say in the previous:
"The original idea is that we can distinguish different state store
exception for different handling. But to be honest, I am not quite sure
this is necessary. Maybe have some change during implementation."
During the implementation, I also feel we maybe not need wrapper it.
We can just throw the correctly directly.
> Is `StreamThreadNotRunningException` really an retryable error?
When KafkaStream state is REBALANCING, I think it is a retryable error.
StreamThreadStateStoreProvider#stores() will throw
StreamThreadNotRunningException when StreamThread state is not RUNNING. The
user can retry until KafkaStream state is RUNNING.
> When would we throw an `StateStoreEmptyException`? The semantics is
unclear to me atm.
> When the state is RUNNING, is `StateStoreClosedException` a retryable
error?
These two comments will be answered in another mail.
---
Vito
On Mon, Jun 11, 2018 at 8:12 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Vito,
>
> what is the status of this KIP?
>
> I just re-read it, and have a couple of follow up comments. Why do we
> discuss the internal exceptions you want to add? Also, do we really need
> them? Can't we just throw the correct exception directly instead of
> wrapping it later?
>
> When would we throw an `StateStoreEmptyException`? The semantics is
> unclear to me atm.
>
> Is `StreamThreadNotRunningException` really an retryable error?
>
> When the state is RUNNING, is `StateStoreClosedException` a retryable
> error?
>
> One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
> time); that should be added
>
>
> Overall I like the KIP but some details are still unclear. Maybe it
> might help if you open an PR in parallel?
>
>
> -Matthias
>
> On 4/24/18 8:18 AM, vito jeng wrote:
> > Hi, Guozhang,
> >
> > Thanks for the comment!
> >
> >
> >
> > Hi, Bill,
> >
> > I'll try to make some update to make the KIP better.
> >
> > Thanks for the comment!
> >
> >
> > ---
> > Vito
> >
> > On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com> wrote:
> >
> >> Hi Vito,
> >>
> >> Thanks for the KIP, overall it's a +1 from me.
> >>
> >> At this point, the only thing I would change is possibly removing the
> >> listing of all methods called by the user and the listing of all store
> >> types and focus on what states result in which exceptions thrown to the
> >> user.
> >>
> >> Thanks,
> >> Bill
> >>
> >> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >>
> >>> Thanks for the KIP Vito!
> >>>
> >>> I made a pass over the wiki and it looks great to me. I'm +1 on the
> KIP.
> >>>
> >>> About the base class InvalidStateStoreException itself, I'd actually
> >>> suggest we do not deprecate it but still expose it as part of the
> public
> >>> API, for people who do not want to handle these cases differently (if
> we
> >>> deprecate it then we are enforcing them to capture all three exceptions
> >>> one-by-one).
> >>>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io>
> wrote:
> >>>
> >>>> Hi Vito,
> >>>>
> >>>> Thanks for the KIP!
> >>>>
> >>>> I think it's much nicer to give callers different exceptions to tell
> >> them
> >>>> whether the state store got migrated, whether it's still initializing,
> >> or
> >>>> whether there's some unrecoverable error.
> >>>>
> >>>> In the KIP, it's typically not necessary to discuss non-user-facing
> >>> details
> >>>> such as what exceptions we will throw internally. The KIP is primarily
> >> to
> >>>> discuss public interface changes.
> >>>>
> >>>> You might consider simply removing all the internal details from the
> >> KIP,
> >>>> which will have the dual advantage that it makes the KIP smaller and
> >>> easier
> >>>> to agree on, as well as giving you more freedom in the internal
> details
> >>>> when it comes to implementation.
> >>>>
> >>>> I like your decision to have your refined exceptions extend
> >>>> InvalidStateStoreException to ensure backward compatibility. Since we
> >>> want
> >>>> to encourage callers to catch the more specific exceptions, and we
> >> don't
> >>>> expect to ever throw a raw InvalidStateStoreException anymore, you
> >> might
> >>>> consider adding the @Deprecated annotation to
> >> InvalidStateStoreException.
> >>>> This will gently encourage callers to migrate to the new exception and
> >>> open
> >>>> the possibility of removing InvalidStateStoreException entirely in a
> >>> future
> >>>> release.
> >>>>
> >>>> Thanks,
> >>>> -John
> >>>>
> >>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> >> matthias@confluent.io>
> >>>> wrote:
> >>>>
> >>>>> Thanks for clarification! That makes sense to me.
> >>>>>
> >>>>> Can you update the KIP to make those suggestions explicit?
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 4/18/18 2:19 PM, vito jeng wrote:
> >>>>>> Matthias,
> >>>>>>
> >>>>>> Thanks for the feedback!
> >>>>>>
> >>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>
> >>>>>> Got it!
> >>>>>>
> >>>>>>> The (incomplete) question was, if we need
> >> `StateStoreFailException`
> >>> or
> >>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>> suggest
> >>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >> but
> >>>> only
> >>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>
> >>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown at
> >> all
> >>>>>> anymore,
> >>>>>> but only new sub-classes.
> >>>>>>
> >>>>>>> Not sure what this sentence means:
> >>>>>>>> The internal exception will be wrapped as category exception
> >>> finally.
> >>>>>>> Can you elaborate?
> >>>>>>
> >>>>>> For example, `StreamThreadStateStoreProvider#stores()` will throw
> >>>>>> `StreamThreadNotRunningException`(internal exception).
> >>>>>> And then the internal exception will be wrapped as
> >>>>>> `StateStoreRetryableException` or `StateStoreFailException` during
> >>> the
> >>>>>> `KafkaStreams.store()` and throw to the user.
> >>>>>>
> >>>>>>
> >>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>> unclear
> >>>>>> to me atm why they are introduced.
> >>>>>>
> >>>>>> Hmmm...the purpose of the "internal exceptions" is to distinguish
> >>>> between
> >>>>>> the different kinds of InvalidStateStoreException.
> >>>>>> The original idea is that we can distinguish different state store
> >>>>>> exception for
> >>>>>> different handling.
> >>>>>> But to be honest, I am not quite sure this is necessary. Maybe have
> >>>>>> some change during implementation.
> >>>>>>
> >>>>>> Does it make sense?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> >>>> matthias@confluent.io>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for the update Vito!
> >>>>>>>
> >>>>>>> It's up to you to keep the details part in the KIP or not.
> >>>>>>>
> >>>>>>>
> >>>>>>> The (incomplete) question was, if we need
> >> `StateStoreFailException`
> >>> or
> >>>>>>> if existing `InvalidStateStoreException` could be used? Do you
> >>> suggest
> >>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
> >> but
> >>>> only
> >>>>>>> the new sub-classes (just to get a better understanding).
> >>>>>>>
> >>>>>>>
> >>>>>>> Not sure what this sentence means:
> >>>>>>>
> >>>>>>>> The internal exception will be wrapped as category exception
> >>> finally.
> >>>>>>>
> >>>>>>> Can you elaborate?
> >>>>>>>
> >>>>>>>
> >>>>>>> Can you explain the purpose of the "internal exceptions". It's
> >>> unclear
> >>>>>>> to me atm why they are introduced.
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
> >>>>>>>> Matthias,
> >>>>>>>>
> >>>>>>>> Thanks for the review.
> >>>>>>>> I reply separately in the following sections.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Vito
> >>>>>>>>
> >>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> >>>> matthias@confluent.io
> >>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
> >>>>>>>>>
> >>>>>>>>> Seems you did a very thorough investigation of the code. It's
> >>> useful
> >>>>> to
> >>>>>>>>> understand what user facing interfaces are affected.
> >>>>>>>>
> >>>>>>>> (Some parts might be even too detailed for a KIP.)
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> I also think too detailed. Especially the section `Changes in
> >> call
> >>>>>>> trace`.
> >>>>>>>> Do you think it should be removed?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> To summarize my current understanding of your KIP, the main
> >> change
> >>>> is
> >>>>> to
> >>>>>>>>> introduce new exceptions that extend
> >> `InvalidStateStoreException`.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> yep. Keep compatibility in this KIP is important things.
> >>>>>>>> I think the best way is that all new exceptions extend from
> >>>>>>>> `InvalidStateStoreException`.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Some questions:
> >>>>>>>>>
> >>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException` be
> >> used
> >>>> for
> >>>>>>>>> this purpose?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> Does this question miss some word?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
> >>>> ngException`
> >>>>>>>>> is? Should it be `InvalidStateStoreException` or
> >>>>>>> `StateStoreFailException`
> >>>>>>>>> ?
> >>>>>>>>>
> >>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> I apologize for not well written parts. I tried to modify some
> >> code
> >>>> in
> >>>>>>> the
> >>>>>>>> recent period and modify the KIP.
> >>>>>>>> The modification is now complete. Please look again.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>>>>>>> Matthias,
> >>>>>>>>>>
> >>>>>>>>>> Sorry for not response these days.
> >>>>>>>>>> I just finished it. Please have a look. :)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> Vito
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >>>>>>> matthias@confluent.io>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Is there any update on this KIP?
> >>>>>>>>>>>
> >>>>>>>>>>> -Matthias
> >>>>>>>>>>>
> >>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thank you for your response.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think you are right. We need to look at the state both of
> >>>>>>>>>>>> KafkaStreams and StreamThread.
> >>>>>>>>>>>>
> >>>>>>>>>>>> After further understanding of KafkaStreams thread and state
> >>>> store,
> >>>>>>>>>>>> I am currently rewriting the KIP.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Vito,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sorry for this late reply.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> There can be two cases:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - either a store got migrated way and thus, is not hosted
> >> an
> >>>> the
> >>>>>>>>>>>>> application instance anymore,
> >>>>>>>>>>>>> - or, a store is hosted but the instance is in state
> >>> rebalance
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> For the first case, users need to rediscover the store. For
> >>> the
> >>>>>>> second
> >>>>>>>>>>>>> case, they need to wait until rebalance is finished.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> >>>>> NOT_RUNNING,
> >>>>>>>>>>>>> uses cannot query at all and thus they cannot rediscover or
> >>>> retry.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Does this make sense?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>>>>>>> Matthias,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I try to clarify some concept.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When streams state is REBALANCING, it means the user can
> >> just
> >>>>> plain
> >>>>>>>>>>>>> retry.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> >>> NOT_RUNNING,
> >>>>> it
> >>>>>>>>>>> means
> >>>>>>>>>>>>>> state store migrated to another instance, the user needs to
> >>>>>>>>> rediscover
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>> store.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is my understanding correct?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>>>>>>>>> matthias@confluent.io>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of the
> >>> Jira
> >>>>>>> was,
> >>>>>>>>> to
> >>>>>>>>>>>>>>> give different exceptions for different "recovery"
> >>> strategies
> >>>> to
> >>>>>>> the
> >>>>>>>>>>>>> user.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For example, if a store is currently recreated, a user
> >> just
> >>>> need
> >>>>>>> to
> >>>>>>>>>>> wait
> >>>>>>>>>>>>>>> and can query the store later. On the other hand, if a
> >> store
> >>>> go
> >>>>>>>>>>> migrated
> >>>>>>>>>>>>>>> to another instance, a user needs to rediscover the store
> >>>>> instead
> >>>>>>>>> of a
> >>>>>>>>>>>>>>> "plain retry".
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks about
> >>> this
> >>>>>>> ideas
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> reasoning.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
> >> first
> >>>> is
> >>>>>>> what
> >>>>>>>>>>>>>>>> categories of errors we want to notify the users, if we
> >>> only
> >>>>>>> wants
> >>>>>>>>> to
> >>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
> >>>> rename
> >>>>>>> the
> >>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
> >>>>> StateStoreClosedException
> >>>>>>>>>>>>> classes.
> >>>>>>>>>>>>>>>> And then from there we should list what are the possible
> >>>>> internal
> >>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call trace,
> >> and
> >>>>> which
> >>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and which
> >> ones
> >>>>>>> should
> >>>>>>>>> be
> >>>>>>>>>>>>>>>> handled without re-throwing, and which ones should not be
> >>>>> wrapped
> >>>>>>>>> at
> >>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Guozhang
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> >>>>> vito@is-land.com.tw>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> >>>>>>> different+errors
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>> Vito
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Vito,
what is the status of this KIP?
I just re-read it, and have a couple of follow up comments. Why do we
discuss the internal exceptions you want to add? Also, do we really need
them? Can't we just throw the correct exception directly instead of
wrapping it later?
When would we throw an `StateStoreEmptyException`? The semantics is
unclear to me atm.
Is `StreamThreadNotRunningException` really an retryable error?
When the state is RUNNING, is `StateStoreClosedException` a retryable error?
One more nits: ReadOnlyWindowStore got a new method #fetch(K key, long
time); that should be added
Overall I like the KIP but some details are still unclear. Maybe it
might help if you open an PR in parallel?
-Matthias
On 4/24/18 8:18 AM, vito jeng wrote:
> Hi, Guozhang,
>
> Thanks for the comment!
>
>
>
> Hi, Bill,
>
> I'll try to make some update to make the KIP better.
>
> Thanks for the comment!
>
>
> ---
> Vito
>
> On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com> wrote:
>
>> Hi Vito,
>>
>> Thanks for the KIP, overall it's a +1 from me.
>>
>> At this point, the only thing I would change is possibly removing the
>> listing of all methods called by the user and the listing of all store
>> types and focus on what states result in which exceptions thrown to the
>> user.
>>
>> Thanks,
>> Bill
>>
>> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>>
>>> Thanks for the KIP Vito!
>>>
>>> I made a pass over the wiki and it looks great to me. I'm +1 on the KIP.
>>>
>>> About the base class InvalidStateStoreException itself, I'd actually
>>> suggest we do not deprecate it but still expose it as part of the public
>>> API, for people who do not want to handle these cases differently (if we
>>> deprecate it then we are enforcing them to capture all three exceptions
>>> one-by-one).
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io> wrote:
>>>
>>>> Hi Vito,
>>>>
>>>> Thanks for the KIP!
>>>>
>>>> I think it's much nicer to give callers different exceptions to tell
>> them
>>>> whether the state store got migrated, whether it's still initializing,
>> or
>>>> whether there's some unrecoverable error.
>>>>
>>>> In the KIP, it's typically not necessary to discuss non-user-facing
>>> details
>>>> such as what exceptions we will throw internally. The KIP is primarily
>> to
>>>> discuss public interface changes.
>>>>
>>>> You might consider simply removing all the internal details from the
>> KIP,
>>>> which will have the dual advantage that it makes the KIP smaller and
>>> easier
>>>> to agree on, as well as giving you more freedom in the internal details
>>>> when it comes to implementation.
>>>>
>>>> I like your decision to have your refined exceptions extend
>>>> InvalidStateStoreException to ensure backward compatibility. Since we
>>> want
>>>> to encourage callers to catch the more specific exceptions, and we
>> don't
>>>> expect to ever throw a raw InvalidStateStoreException anymore, you
>> might
>>>> consider adding the @Deprecated annotation to
>> InvalidStateStoreException.
>>>> This will gently encourage callers to migrate to the new exception and
>>> open
>>>> the possibility of removing InvalidStateStoreException entirely in a
>>> future
>>>> release.
>>>>
>>>> Thanks,
>>>> -John
>>>>
>>>> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
>> matthias@confluent.io>
>>>> wrote:
>>>>
>>>>> Thanks for clarification! That makes sense to me.
>>>>>
>>>>> Can you update the KIP to make those suggestions explicit?
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 4/18/18 2:19 PM, vito jeng wrote:
>>>>>> Matthias,
>>>>>>
>>>>>> Thanks for the feedback!
>>>>>>
>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>
>>>>>> Got it!
>>>>>>
>>>>>>> The (incomplete) question was, if we need
>> `StateStoreFailException`
>>> or
>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
>>> suggest
>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
>> but
>>>> only
>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>
>>>>>> Yes. I suggest that `InvalidStateStoreException` is not thrown at
>> all
>>>>>> anymore,
>>>>>> but only new sub-classes.
>>>>>>
>>>>>>> Not sure what this sentence means:
>>>>>>>> The internal exception will be wrapped as category exception
>>> finally.
>>>>>>> Can you elaborate?
>>>>>>
>>>>>> For example, `StreamThreadStateStoreProvider#stores()` will throw
>>>>>> `StreamThreadNotRunningException`(internal exception).
>>>>>> And then the internal exception will be wrapped as
>>>>>> `StateStoreRetryableException` or `StateStoreFailException` during
>>> the
>>>>>> `KafkaStreams.store()` and throw to the user.
>>>>>>
>>>>>>
>>>>>>> Can you explain the purpose of the "internal exceptions". It's
>>> unclear
>>>>>> to me atm why they are introduced.
>>>>>>
>>>>>> Hmmm...the purpose of the "internal exceptions" is to distinguish
>>>> between
>>>>>> the different kinds of InvalidStateStoreException.
>>>>>> The original idea is that we can distinguish different state store
>>>>>> exception for
>>>>>> different handling.
>>>>>> But to be honest, I am not quite sure this is necessary. Maybe have
>>>>>> some change during implementation.
>>>>>>
>>>>>> Does it make sense?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Vito
>>>>>>
>>>>>> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
>>>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the update Vito!
>>>>>>>
>>>>>>> It's up to you to keep the details part in the KIP or not.
>>>>>>>
>>>>>>>
>>>>>>> The (incomplete) question was, if we need
>> `StateStoreFailException`
>>> or
>>>>>>> if existing `InvalidStateStoreException` could be used? Do you
>>> suggest
>>>>>>> that `InvalidStateStoreException` is not thrown at all anymore,
>> but
>>>> only
>>>>>>> the new sub-classes (just to get a better understanding).
>>>>>>>
>>>>>>>
>>>>>>> Not sure what this sentence means:
>>>>>>>
>>>>>>>> The internal exception will be wrapped as category exception
>>> finally.
>>>>>>>
>>>>>>> Can you elaborate?
>>>>>>>
>>>>>>>
>>>>>>> Can you explain the purpose of the "internal exceptions". It's
>>> unclear
>>>>>>> to me atm why they are introduced.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>> On 4/10/18 12:33 AM, vito jeng wrote:
>>>>>>>> Matthias,
>>>>>>>>
>>>>>>>> Thanks for the review.
>>>>>>>> I reply separately in the following sections.
>>>>>>>>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
>>>> matthias@confluent.io
>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for updating the KIP and sorry for the long pause...
>>>>>>>>>
>>>>>>>>> Seems you did a very thorough investigation of the code. It's
>>> useful
>>>>> to
>>>>>>>>> understand what user facing interfaces are affected.
>>>>>>>>
>>>>>>>> (Some parts might be even too detailed for a KIP.)
>>>>>>>>>
>>>>>>>>
>>>>>>>> I also think too detailed. Especially the section `Changes in
>> call
>>>>>>> trace`.
>>>>>>>> Do you think it should be removed?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> To summarize my current understanding of your KIP, the main
>> change
>>>> is
>>>>> to
>>>>>>>>> introduce new exceptions that extend
>> `InvalidStateStoreException`.
>>>>>>>>>
>>>>>>>>
>>>>>>>> yep. Keep compatibility in this KIP is important things.
>>>>>>>> I think the best way is that all new exceptions extend from
>>>>>>>> `InvalidStateStoreException`.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Some questions:
>>>>>>>>>
>>>>>>>>> - Why do we need ```? Could `InvalidStateStoreException` be
>> used
>>>> for
>>>>>>>>> this purpose?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Does this question miss some word?
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> - What the superclass of `StateStoreStreamThreadNotRunni
>>>> ngException`
>>>>>>>>> is? Should it be `InvalidStateStoreException` or
>>>>>>> `StateStoreFailException`
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I apologize for not well written parts. I tried to modify some
>> code
>>>> in
>>>>>>> the
>>>>>>>> recent period and modify the KIP.
>>>>>>>> The modification is now complete. Please look again.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>>>>>>> Matthias,
>>>>>>>>>>
>>>>>>>>>> Sorry for not response these days.
>>>>>>>>>> I just finished it. Please have a look. :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>>>>>>> matthias@confluent.io>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Is there any update on this KIP?
>>>>>>>>>>>
>>>>>>>>>>> -Matthias
>>>>>>>>>>>
>>>>>>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for your response.
>>>>>>>>>>>>
>>>>>>>>>>>> I think you are right. We need to look at the state both of
>>>>>>>>>>>> KafkaStreams and StreamThread.
>>>>>>>>>>>>
>>>>>>>>>>>> After further understanding of KafkaStreams thread and state
>>>> store,
>>>>>>>>>>>> I am currently rewriting the KIP.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Vito
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Vito,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry for this late reply.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There can be two cases:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - either a store got migrated way and thus, is not hosted
>> an
>>>> the
>>>>>>>>>>>>> application instance anymore,
>>>>>>>>>>>>> - or, a store is hosted but the instance is in state
>>> rebalance
>>>>>>>>>>>>>
>>>>>>>>>>>>> For the first case, users need to rediscover the store. For
>>> the
>>>>>>> second
>>>>>>>>>>>>> case, they need to wait until rebalance is finished.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
>>>>> NOT_RUNNING,
>>>>>>>>>>>>> uses cannot query at all and thus they cannot rediscover or
>>>> retry.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does this make sense?
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>>>>>>> Matthias,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I try to clarify some concept.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When streams state is REBALANCING, it means the user can
>> just
>>>>> plain
>>>>>>>>>>>>> retry.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
>>> NOT_RUNNING,
>>>>> it
>>>>>>>>>>> means
>>>>>>>>>>>>>> state store migrated to another instance, the user needs to
>>>>>>>>> rediscover
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> store.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is my understanding correct?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>>>>>>>>>>> matthias@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I agree with what Guozhang said. The original idea of the
>>> Jira
>>>>>>> was,
>>>>>>>>> to
>>>>>>>>>>>>>>> give different exceptions for different "recovery"
>>> strategies
>>>> to
>>>>>>> the
>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For example, if a store is currently recreated, a user
>> just
>>>> need
>>>>>>> to
>>>>>>>>>>> wait
>>>>>>>>>>>>>>> and can query the store later. On the other hand, if a
>> store
>>>> go
>>>>>>>>>>> migrated
>>>>>>>>>>>>>>> to another instance, a user needs to rediscover the store
>>>>> instead
>>>>>>>>> of a
>>>>>>>>>>>>>>> "plain retry".
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Anyway, the KIP should contain a section that talks about
>>> this
>>>>>>> ideas
>>>>>>>>>>> and
>>>>>>>>>>>>>>> reasoning.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Matthias
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
>> first
>>>> is
>>>>>>> what
>>>>>>>>>>>>>>>> categories of errors we want to notify the users, if we
>>> only
>>>>>>> wants
>>>>>>>>> to
>>>>>>>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
>>>> rename
>>>>>>> the
>>>>>>>>>>>>>>>> proposed StateStoreMigratedException /
>>>>> StateStoreClosedException
>>>>>>>>>>>>> classes.
>>>>>>>>>>>>>>>> And then from there we should list what are the possible
>>>>> internal
>>>>>>>>>>>>>>>> exceptions ever thrown in those APIs in the call trace,
>> and
>>>>> which
>>>>>>>>>>>>>>>> exceptions should be wrapped to what others, and which
>> ones
>>>>>>> should
>>>>>>>>> be
>>>>>>>>>>>>>>>> handled without re-throwing, and which ones should not be
>>>>> wrapped
>>>>>>>>> at
>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Guozhang
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
>>>>> vito@is-land.com.tw>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>>>>>>> different+errors
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>> Vito
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> -- Guozhang
>>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Hi, Guozhang,
Thanks for the comment!
Hi, Bill,
I'll try to make some update to make the KIP better.
Thanks for the comment!
---
Vito
On Sat, Apr 21, 2018 at 5:40 AM, Bill Bejeck <bb...@gmail.com> wrote:
> Hi Vito,
>
> Thanks for the KIP, overall it's a +1 from me.
>
> At this point, the only thing I would change is possibly removing the
> listing of all methods called by the user and the listing of all store
> types and focus on what states result in which exceptions thrown to the
> user.
>
> Thanks,
> Bill
>
> On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Thanks for the KIP Vito!
> >
> > I made a pass over the wiki and it looks great to me. I'm +1 on the KIP.
> >
> > About the base class InvalidStateStoreException itself, I'd actually
> > suggest we do not deprecate it but still expose it as part of the public
> > API, for people who do not want to handle these cases differently (if we
> > deprecate it then we are enforcing them to capture all three exceptions
> > one-by-one).
> >
> >
> > Guozhang
> >
> >
> > On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io> wrote:
> >
> > > Hi Vito,
> > >
> > > Thanks for the KIP!
> > >
> > > I think it's much nicer to give callers different exceptions to tell
> them
> > > whether the state store got migrated, whether it's still initializing,
> or
> > > whether there's some unrecoverable error.
> > >
> > > In the KIP, it's typically not necessary to discuss non-user-facing
> > details
> > > such as what exceptions we will throw internally. The KIP is primarily
> to
> > > discuss public interface changes.
> > >
> > > You might consider simply removing all the internal details from the
> KIP,
> > > which will have the dual advantage that it makes the KIP smaller and
> > easier
> > > to agree on, as well as giving you more freedom in the internal details
> > > when it comes to implementation.
> > >
> > > I like your decision to have your refined exceptions extend
> > > InvalidStateStoreException to ensure backward compatibility. Since we
> > want
> > > to encourage callers to catch the more specific exceptions, and we
> don't
> > > expect to ever throw a raw InvalidStateStoreException anymore, you
> might
> > > consider adding the @Deprecated annotation to
> InvalidStateStoreException.
> > > This will gently encourage callers to migrate to the new exception and
> > open
> > > the possibility of removing InvalidStateStoreException entirely in a
> > future
> > > release.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <
> matthias@confluent.io>
> > > wrote:
> > >
> > > > Thanks for clarification! That makes sense to me.
> > > >
> > > > Can you update the KIP to make those suggestions explicit?
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 4/18/18 2:19 PM, vito jeng wrote:
> > > > > Matthias,
> > > > >
> > > > > Thanks for the feedback!
> > > > >
> > > > >> It's up to you to keep the details part in the KIP or not.
> > > > >
> > > > > Got it!
> > > > >
> > > > >> The (incomplete) question was, if we need
> `StateStoreFailException`
> > or
> > > > >> if existing `InvalidStateStoreException` could be used? Do you
> > suggest
> > > > >> that `InvalidStateStoreException` is not thrown at all anymore,
> but
> > > only
> > > > >> the new sub-classes (just to get a better understanding).
> > > > >
> > > > > Yes. I suggest that `InvalidStateStoreException` is not thrown at
> all
> > > > > anymore,
> > > > > but only new sub-classes.
> > > > >
> > > > >> Not sure what this sentence means:
> > > > >>> The internal exception will be wrapped as category exception
> > finally.
> > > > >> Can you elaborate?
> > > > >
> > > > > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > > > > `StreamThreadNotRunningException`(internal exception).
> > > > > And then the internal exception will be wrapped as
> > > > > `StateStoreRetryableException` or `StateStoreFailException` during
> > the
> > > > > `KafkaStreams.store()` and throw to the user.
> > > > >
> > > > >
> > > > >> Can you explain the purpose of the "internal exceptions". It's
> > unclear
> > > > > to me atm why they are introduced.
> > > > >
> > > > > Hmmm...the purpose of the "internal exceptions" is to distinguish
> > > between
> > > > > the different kinds of InvalidStateStoreException.
> > > > > The original idea is that we can distinguish different state store
> > > > > exception for
> > > > > different handling.
> > > > > But to be honest, I am not quite sure this is necessary. Maybe have
> > > > > some change during implementation.
> > > > >
> > > > > Does it make sense?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ---
> > > > > Vito
> > > > >
> > > > > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> > > matthias@confluent.io>
> > > > > wrote:
> > > > >
> > > > >> Thanks for the update Vito!
> > > > >>
> > > > >> It's up to you to keep the details part in the KIP or not.
> > > > >>
> > > > >>
> > > > >> The (incomplete) question was, if we need
> `StateStoreFailException`
> > or
> > > > >> if existing `InvalidStateStoreException` could be used? Do you
> > suggest
> > > > >> that `InvalidStateStoreException` is not thrown at all anymore,
> but
> > > only
> > > > >> the new sub-classes (just to get a better understanding).
> > > > >>
> > > > >>
> > > > >> Not sure what this sentence means:
> > > > >>
> > > > >>> The internal exception will be wrapped as category exception
> > finally.
> > > > >>
> > > > >> Can you elaborate?
> > > > >>
> > > > >>
> > > > >> Can you explain the purpose of the "internal exceptions". It's
> > unclear
> > > > >> to me atm why they are introduced.
> > > > >>
> > > > >>
> > > > >> -Matthias
> > > > >>
> > > > >> On 4/10/18 12:33 AM, vito jeng wrote:
> > > > >>> Matthias,
> > > > >>>
> > > > >>> Thanks for the review.
> > > > >>> I reply separately in the following sections.
> > > > >>>
> > > > >>>
> > > > >>> ---
> > > > >>> Vito
> > > > >>>
> > > > >>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> > > matthias@confluent.io
> > > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Thanks for updating the KIP and sorry for the long pause...
> > > > >>>>
> > > > >>>> Seems you did a very thorough investigation of the code. It's
> > useful
> > > > to
> > > > >>>> understand what user facing interfaces are affected.
> > > > >>>
> > > > >>> (Some parts might be even too detailed for a KIP.)
> > > > >>>>
> > > > >>>
> > > > >>> I also think too detailed. Especially the section `Changes in
> call
> > > > >> trace`.
> > > > >>> Do you think it should be removed?
> > > > >>>
> > > > >>>
> > > > >>>>
> > > > >>>> To summarize my current understanding of your KIP, the main
> change
> > > is
> > > > to
> > > > >>>> introduce new exceptions that extend
> `InvalidStateStoreException`.
> > > > >>>>
> > > > >>>
> > > > >>> yep. Keep compatibility in this KIP is important things.
> > > > >>> I think the best way is that all new exceptions extend from
> > > > >>> `InvalidStateStoreException`.
> > > > >>>
> > > > >>>
> > > > >>>>
> > > > >>>> Some questions:
> > > > >>>>
> > > > >>>> - Why do we need ```? Could `InvalidStateStoreException` be
> used
> > > for
> > > > >>>> this purpose?
> > > > >>>>
> > > > >>>
> > > > >>> Does this question miss some word?
> > > > >>>
> > > > >>>
> > > > >>>>
> > > > >>>> - What the superclass of `StateStoreStreamThreadNotRunni
> > > ngException`
> > > > >>>> is? Should it be `InvalidStateStoreException` or
> > > > >> `StateStoreFailException`
> > > > >>>> ?
> > > > >>>>
> > > > >>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> > > > >>>>
> > > > >>>>
> > > > >>> I apologize for not well written parts. I tried to modify some
> code
> > > in
> > > > >> the
> > > > >>> recent period and modify the KIP.
> > > > >>> The modification is now complete. Please look again.
> > > > >>>
> > > > >>>
> > > > >>>>
> > > > >>>>
> > > > >>>> -Matthias
> > > > >>>>
> > > > >>>>
> > > > >>>> On 2/21/18 5:15 PM, vito jeng wrote:
> > > > >>>>> Matthias,
> > > > >>>>>
> > > > >>>>> Sorry for not response these days.
> > > > >>>>> I just finished it. Please have a look. :)
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> ---
> > > > >>>>> Vito
> > > > >>>>>
> > > > >>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> > > > >> matthias@confluent.io>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Is there any update on this KIP?
> > > > >>>>>>
> > > > >>>>>> -Matthias
> > > > >>>>>>
> > > > >>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> > > > >>>>>>> Matthias,
> > > > >>>>>>>
> > > > >>>>>>> Thank you for your response.
> > > > >>>>>>>
> > > > >>>>>>> I think you are right. We need to look at the state both of
> > > > >>>>>>> KafkaStreams and StreamThread.
> > > > >>>>>>>
> > > > >>>>>>> After further understanding of KafkaStreams thread and state
> > > store,
> > > > >>>>>>> I am currently rewriting the KIP.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> ---
> > > > >>>>>>> Vito
> > > > >>>>>>>
> > > > >>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> > > > >>>> matthias@confluent.io>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Vito,
> > > > >>>>>>>>
> > > > >>>>>>>> Sorry for this late reply.
> > > > >>>>>>>>
> > > > >>>>>>>> There can be two cases:
> > > > >>>>>>>>
> > > > >>>>>>>> - either a store got migrated way and thus, is not hosted
> an
> > > the
> > > > >>>>>>>> application instance anymore,
> > > > >>>>>>>> - or, a store is hosted but the instance is in state
> > rebalance
> > > > >>>>>>>>
> > > > >>>>>>>> For the first case, users need to rediscover the store. For
> > the
> > > > >> second
> > > > >>>>>>>> case, they need to wait until rebalance is finished.
> > > > >>>>>>>>
> > > > >>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> > > > NOT_RUNNING,
> > > > >>>>>>>> uses cannot query at all and thus they cannot rediscover or
> > > retry.
> > > > >>>>>>>>
> > > > >>>>>>>> Does this make sense?
> > > > >>>>>>>>
> > > > >>>>>>>> -Matthias
> > > > >>>>>>>>
> > > > >>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> > > > >>>>>>>>> Matthias,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I try to clarify some concept.
> > > > >>>>>>>>>
> > > > >>>>>>>>> When streams state is REBALANCING, it means the user can
> just
> > > > plain
> > > > >>>>>>>> retry.
> > > > >>>>>>>>>
> > > > >>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> > NOT_RUNNING,
> > > > it
> > > > >>>>>> means
> > > > >>>>>>>>> state store migrated to another instance, the user needs to
> > > > >>>> rediscover
> > > > >>>>>>>> the
> > > > >>>>>>>>> store.
> > > > >>>>>>>>>
> > > > >>>>>>>>> Is my understanding correct?
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> ---
> > > > >>>>>>>>> Vito
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> > > > >>>>>> matthias@confluent.io>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Thanks for the KIP Vito!
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I agree with what Guozhang said. The original idea of the
> > Jira
> > > > >> was,
> > > > >>>> to
> > > > >>>>>>>>>> give different exceptions for different "recovery"
> > strategies
> > > to
> > > > >> the
> > > > >>>>>>>> user.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> For example, if a store is currently recreated, a user
> just
> > > need
> > > > >> to
> > > > >>>>>> wait
> > > > >>>>>>>>>> and can query the store later. On the other hand, if a
> store
> > > go
> > > > >>>>>> migrated
> > > > >>>>>>>>>> to another instance, a user needs to rediscover the store
> > > > instead
> > > > >>>> of a
> > > > >>>>>>>>>> "plain retry".
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Fatal errors might be a third category.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Not sure if there is something else?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Anyway, the KIP should contain a section that talks about
> > this
> > > > >> ideas
> > > > >>>>>> and
> > > > >>>>>>>>>> reasoning.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> -Matthias
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > > > >>>>>>>>>>> Thanks for writing up the KIP.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out
> first
> > > is
> > > > >> what
> > > > >>>>>>>>>>> categories of errors we want to notify the users, if we
> > only
> > > > >> wants
> > > > >>>> to
> > > > >>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
> > > rename
> > > > >> the
> > > > >>>>>>>>>>> proposed StateStoreMigratedException /
> > > > StateStoreClosedException
> > > > >>>>>>>> classes.
> > > > >>>>>>>>>>> And then from there we should list what are the possible
> > > > internal
> > > > >>>>>>>>>>> exceptions ever thrown in those APIs in the call trace,
> and
> > > > which
> > > > >>>>>>>>>>> exceptions should be wrapped to what others, and which
> ones
> > > > >> should
> > > > >>>> be
> > > > >>>>>>>>>>> handled without re-throwing, and which ones should not be
> > > > wrapped
> > > > >>>> at
> > > > >>>>>>>> all
> > > > >>>>>>>>>>> but directly thrown to user's face.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Guozhang
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> > > > vito@is-land.com.tw>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I'd like to start discuss KIP-216:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> > > > >> different+errors
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Please have a look.
> > > > >>>>>>>>>>>> Thanks!
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> ---
> > > > >>>>>>>>>>>> Vito
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Bill Bejeck <bb...@gmail.com>.
Hi Vito,
Thanks for the KIP, overall it's a +1 from me.
At this point, the only thing I would change is possibly removing the
listing of all methods called by the user and the listing of all store
types and focus on what states result in which exceptions thrown to the
user.
Thanks,
Bill
On Fri, Apr 20, 2018 at 2:10 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Thanks for the KIP Vito!
>
> I made a pass over the wiki and it looks great to me. I'm +1 on the KIP.
>
> About the base class InvalidStateStoreException itself, I'd actually
> suggest we do not deprecate it but still expose it as part of the public
> API, for people who do not want to handle these cases differently (if we
> deprecate it then we are enforcing them to capture all three exceptions
> one-by-one).
>
>
> Guozhang
>
>
> On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io> wrote:
>
> > Hi Vito,
> >
> > Thanks for the KIP!
> >
> > I think it's much nicer to give callers different exceptions to tell them
> > whether the state store got migrated, whether it's still initializing, or
> > whether there's some unrecoverable error.
> >
> > In the KIP, it's typically not necessary to discuss non-user-facing
> details
> > such as what exceptions we will throw internally. The KIP is primarily to
> > discuss public interface changes.
> >
> > You might consider simply removing all the internal details from the KIP,
> > which will have the dual advantage that it makes the KIP smaller and
> easier
> > to agree on, as well as giving you more freedom in the internal details
> > when it comes to implementation.
> >
> > I like your decision to have your refined exceptions extend
> > InvalidStateStoreException to ensure backward compatibility. Since we
> want
> > to encourage callers to catch the more specific exceptions, and we don't
> > expect to ever throw a raw InvalidStateStoreException anymore, you might
> > consider adding the @Deprecated annotation to InvalidStateStoreException.
> > This will gently encourage callers to migrate to the new exception and
> open
> > the possibility of removing InvalidStateStoreException entirely in a
> future
> > release.
> >
> > Thanks,
> > -John
> >
> > On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> > > Thanks for clarification! That makes sense to me.
> > >
> > > Can you update the KIP to make those suggestions explicit?
> > >
> > >
> > > -Matthias
> > >
> > > On 4/18/18 2:19 PM, vito jeng wrote:
> > > > Matthias,
> > > >
> > > > Thanks for the feedback!
> > > >
> > > >> It's up to you to keep the details part in the KIP or not.
> > > >
> > > > Got it!
> > > >
> > > >> The (incomplete) question was, if we need `StateStoreFailException`
> or
> > > >> if existing `InvalidStateStoreException` could be used? Do you
> suggest
> > > >> that `InvalidStateStoreException` is not thrown at all anymore, but
> > only
> > > >> the new sub-classes (just to get a better understanding).
> > > >
> > > > Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> > > > anymore,
> > > > but only new sub-classes.
> > > >
> > > >> Not sure what this sentence means:
> > > >>> The internal exception will be wrapped as category exception
> finally.
> > > >> Can you elaborate?
> > > >
> > > > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > > > `StreamThreadNotRunningException`(internal exception).
> > > > And then the internal exception will be wrapped as
> > > > `StateStoreRetryableException` or `StateStoreFailException` during
> the
> > > > `KafkaStreams.store()` and throw to the user.
> > > >
> > > >
> > > >> Can you explain the purpose of the "internal exceptions". It's
> unclear
> > > > to me atm why they are introduced.
> > > >
> > > > Hmmm...the purpose of the "internal exceptions" is to distinguish
> > between
> > > > the different kinds of InvalidStateStoreException.
> > > > The original idea is that we can distinguish different state store
> > > > exception for
> > > > different handling.
> > > > But to be honest, I am not quite sure this is necessary. Maybe have
> > > > some change during implementation.
> > > >
> > > > Does it make sense?
> > > >
> > > >
> > > >
> > > >
> > > > ---
> > > > Vito
> > > >
> > > > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> > matthias@confluent.io>
> > > > wrote:
> > > >
> > > >> Thanks for the update Vito!
> > > >>
> > > >> It's up to you to keep the details part in the KIP or not.
> > > >>
> > > >>
> > > >> The (incomplete) question was, if we need `StateStoreFailException`
> or
> > > >> if existing `InvalidStateStoreException` could be used? Do you
> suggest
> > > >> that `InvalidStateStoreException` is not thrown at all anymore, but
> > only
> > > >> the new sub-classes (just to get a better understanding).
> > > >>
> > > >>
> > > >> Not sure what this sentence means:
> > > >>
> > > >>> The internal exception will be wrapped as category exception
> finally.
> > > >>
> > > >> Can you elaborate?
> > > >>
> > > >>
> > > >> Can you explain the purpose of the "internal exceptions". It's
> unclear
> > > >> to me atm why they are introduced.
> > > >>
> > > >>
> > > >> -Matthias
> > > >>
> > > >> On 4/10/18 12:33 AM, vito jeng wrote:
> > > >>> Matthias,
> > > >>>
> > > >>> Thanks for the review.
> > > >>> I reply separately in the following sections.
> > > >>>
> > > >>>
> > > >>> ---
> > > >>> Vito
> > > >>>
> > > >>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> > matthias@confluent.io
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> Thanks for updating the KIP and sorry for the long pause...
> > > >>>>
> > > >>>> Seems you did a very thorough investigation of the code. It's
> useful
> > > to
> > > >>>> understand what user facing interfaces are affected.
> > > >>>
> > > >>> (Some parts might be even too detailed for a KIP.)
> > > >>>>
> > > >>>
> > > >>> I also think too detailed. Especially the section `Changes in call
> > > >> trace`.
> > > >>> Do you think it should be removed?
> > > >>>
> > > >>>
> > > >>>>
> > > >>>> To summarize my current understanding of your KIP, the main change
> > is
> > > to
> > > >>>> introduce new exceptions that extend `InvalidStateStoreException`.
> > > >>>>
> > > >>>
> > > >>> yep. Keep compatibility in this KIP is important things.
> > > >>> I think the best way is that all new exceptions extend from
> > > >>> `InvalidStateStoreException`.
> > > >>>
> > > >>>
> > > >>>>
> > > >>>> Some questions:
> > > >>>>
> > > >>>> - Why do we need ```? Could `InvalidStateStoreException` be used
> > for
> > > >>>> this purpose?
> > > >>>>
> > > >>>
> > > >>> Does this question miss some word?
> > > >>>
> > > >>>
> > > >>>>
> > > >>>> - What the superclass of `StateStoreStreamThreadNotRunni
> > ngException`
> > > >>>> is? Should it be `InvalidStateStoreException` or
> > > >> `StateStoreFailException`
> > > >>>> ?
> > > >>>>
> > > >>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> > > >>>>
> > > >>>>
> > > >>> I apologize for not well written parts. I tried to modify some code
> > in
> > > >> the
> > > >>> recent period and modify the KIP.
> > > >>> The modification is now complete. Please look again.
> > > >>>
> > > >>>
> > > >>>>
> > > >>>>
> > > >>>> -Matthias
> > > >>>>
> > > >>>>
> > > >>>> On 2/21/18 5:15 PM, vito jeng wrote:
> > > >>>>> Matthias,
> > > >>>>>
> > > >>>>> Sorry for not response these days.
> > > >>>>> I just finished it. Please have a look. :)
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> ---
> > > >>>>> Vito
> > > >>>>>
> > > >>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> > > >> matthias@confluent.io>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Is there any update on this KIP?
> > > >>>>>>
> > > >>>>>> -Matthias
> > > >>>>>>
> > > >>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> > > >>>>>>> Matthias,
> > > >>>>>>>
> > > >>>>>>> Thank you for your response.
> > > >>>>>>>
> > > >>>>>>> I think you are right. We need to look at the state both of
> > > >>>>>>> KafkaStreams and StreamThread.
> > > >>>>>>>
> > > >>>>>>> After further understanding of KafkaStreams thread and state
> > store,
> > > >>>>>>> I am currently rewriting the KIP.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> ---
> > > >>>>>>> Vito
> > > >>>>>>>
> > > >>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> > > >>>> matthias@confluent.io>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Vito,
> > > >>>>>>>>
> > > >>>>>>>> Sorry for this late reply.
> > > >>>>>>>>
> > > >>>>>>>> There can be two cases:
> > > >>>>>>>>
> > > >>>>>>>> - either a store got migrated way and thus, is not hosted an
> > the
> > > >>>>>>>> application instance anymore,
> > > >>>>>>>> - or, a store is hosted but the instance is in state
> rebalance
> > > >>>>>>>>
> > > >>>>>>>> For the first case, users need to rediscover the store. For
> the
> > > >> second
> > > >>>>>>>> case, they need to wait until rebalance is finished.
> > > >>>>>>>>
> > > >>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> > > NOT_RUNNING,
> > > >>>>>>>> uses cannot query at all and thus they cannot rediscover or
> > retry.
> > > >>>>>>>>
> > > >>>>>>>> Does this make sense?
> > > >>>>>>>>
> > > >>>>>>>> -Matthias
> > > >>>>>>>>
> > > >>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> > > >>>>>>>>> Matthias,
> > > >>>>>>>>>
> > > >>>>>>>>> I try to clarify some concept.
> > > >>>>>>>>>
> > > >>>>>>>>> When streams state is REBALANCING, it means the user can just
> > > plain
> > > >>>>>>>> retry.
> > > >>>>>>>>>
> > > >>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or
> NOT_RUNNING,
> > > it
> > > >>>>>> means
> > > >>>>>>>>> state store migrated to another instance, the user needs to
> > > >>>> rediscover
> > > >>>>>>>> the
> > > >>>>>>>>> store.
> > > >>>>>>>>>
> > > >>>>>>>>> Is my understanding correct?
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> ---
> > > >>>>>>>>> Vito
> > > >>>>>>>>>
> > > >>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> > > >>>>>> matthias@confluent.io>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Thanks for the KIP Vito!
> > > >>>>>>>>>>
> > > >>>>>>>>>> I agree with what Guozhang said. The original idea of the
> Jira
> > > >> was,
> > > >>>> to
> > > >>>>>>>>>> give different exceptions for different "recovery"
> strategies
> > to
> > > >> the
> > > >>>>>>>> user.
> > > >>>>>>>>>>
> > > >>>>>>>>>> For example, if a store is currently recreated, a user just
> > need
> > > >> to
> > > >>>>>> wait
> > > >>>>>>>>>> and can query the store later. On the other hand, if a store
> > go
> > > >>>>>> migrated
> > > >>>>>>>>>> to another instance, a user needs to rediscover the store
> > > instead
> > > >>>> of a
> > > >>>>>>>>>> "plain retry".
> > > >>>>>>>>>>
> > > >>>>>>>>>> Fatal errors might be a third category.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Not sure if there is something else?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Anyway, the KIP should contain a section that talks about
> this
> > > >> ideas
> > > >>>>>> and
> > > >>>>>>>>>> reasoning.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> -Matthias
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > > >>>>>>>>>>> Thanks for writing up the KIP.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first
> > is
> > > >> what
> > > >>>>>>>>>>> categories of errors we want to notify the users, if we
> only
> > > >> wants
> > > >>>> to
> > > >>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
> > rename
> > > >> the
> > > >>>>>>>>>>> proposed StateStoreMigratedException /
> > > StateStoreClosedException
> > > >>>>>>>> classes.
> > > >>>>>>>>>>> And then from there we should list what are the possible
> > > internal
> > > >>>>>>>>>>> exceptions ever thrown in those APIs in the call trace, and
> > > which
> > > >>>>>>>>>>> exceptions should be wrapped to what others, and which ones
> > > >> should
> > > >>>> be
> > > >>>>>>>>>>> handled without re-throwing, and which ones should not be
> > > wrapped
> > > >>>> at
> > > >>>>>>>> all
> > > >>>>>>>>>>> but directly thrown to user's face.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Guozhang
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> > > vito@is-land.com.tw>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I'd like to start discuss KIP-216:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> > > >> different+errors
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Please have a look.
> > > >>>>>>>>>>>> Thanks!
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> ---
> > > >>>>>>>>>>>> Vito
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Guozhang Wang <wa...@gmail.com>.
Thanks for the KIP Vito!
I made a pass over the wiki and it looks great to me. I'm +1 on the KIP.
About the base class InvalidStateStoreException itself, I'd actually
suggest we do not deprecate it but still expose it as part of the public
API, for people who do not want to handle these cases differently (if we
deprecate it then we are enforcing them to capture all three exceptions
one-by-one).
Guozhang
On Fri, Apr 20, 2018 at 9:14 AM, John Roesler <jo...@confluent.io> wrote:
> Hi Vito,
>
> Thanks for the KIP!
>
> I think it's much nicer to give callers different exceptions to tell them
> whether the state store got migrated, whether it's still initializing, or
> whether there's some unrecoverable error.
>
> In the KIP, it's typically not necessary to discuss non-user-facing details
> such as what exceptions we will throw internally. The KIP is primarily to
> discuss public interface changes.
>
> You might consider simply removing all the internal details from the KIP,
> which will have the dual advantage that it makes the KIP smaller and easier
> to agree on, as well as giving you more freedom in the internal details
> when it comes to implementation.
>
> I like your decision to have your refined exceptions extend
> InvalidStateStoreException to ensure backward compatibility. Since we want
> to encourage callers to catch the more specific exceptions, and we don't
> expect to ever throw a raw InvalidStateStoreException anymore, you might
> consider adding the @Deprecated annotation to InvalidStateStoreException.
> This will gently encourage callers to migrate to the new exception and open
> the possibility of removing InvalidStateStoreException entirely in a future
> release.
>
> Thanks,
> -John
>
> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Thanks for clarification! That makes sense to me.
> >
> > Can you update the KIP to make those suggestions explicit?
> >
> >
> > -Matthias
> >
> > On 4/18/18 2:19 PM, vito jeng wrote:
> > > Matthias,
> > >
> > > Thanks for the feedback!
> > >
> > >> It's up to you to keep the details part in the KIP or not.
> > >
> > > Got it!
> > >
> > >> The (incomplete) question was, if we need `StateStoreFailException` or
> > >> if existing `InvalidStateStoreException` could be used? Do you suggest
> > >> that `InvalidStateStoreException` is not thrown at all anymore, but
> only
> > >> the new sub-classes (just to get a better understanding).
> > >
> > > Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> > > anymore,
> > > but only new sub-classes.
> > >
> > >> Not sure what this sentence means:
> > >>> The internal exception will be wrapped as category exception finally.
> > >> Can you elaborate?
> > >
> > > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > > `StreamThreadNotRunningException`(internal exception).
> > > And then the internal exception will be wrapped as
> > > `StateStoreRetryableException` or `StateStoreFailException` during the
> > > `KafkaStreams.store()` and throw to the user.
> > >
> > >
> > >> Can you explain the purpose of the "internal exceptions". It's unclear
> > > to me atm why they are introduced.
> > >
> > > Hmmm...the purpose of the "internal exceptions" is to distinguish
> between
> > > the different kinds of InvalidStateStoreException.
> > > The original idea is that we can distinguish different state store
> > > exception for
> > > different handling.
> > > But to be honest, I am not quite sure this is necessary. Maybe have
> > > some change during implementation.
> > >
> > > Does it make sense?
> > >
> > >
> > >
> > >
> > > ---
> > > Vito
> > >
> > > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> matthias@confluent.io>
> > > wrote:
> > >
> > >> Thanks for the update Vito!
> > >>
> > >> It's up to you to keep the details part in the KIP or not.
> > >>
> > >>
> > >> The (incomplete) question was, if we need `StateStoreFailException` or
> > >> if existing `InvalidStateStoreException` could be used? Do you suggest
> > >> that `InvalidStateStoreException` is not thrown at all anymore, but
> only
> > >> the new sub-classes (just to get a better understanding).
> > >>
> > >>
> > >> Not sure what this sentence means:
> > >>
> > >>> The internal exception will be wrapped as category exception finally.
> > >>
> > >> Can you elaborate?
> > >>
> > >>
> > >> Can you explain the purpose of the "internal exceptions". It's unclear
> > >> to me atm why they are introduced.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 4/10/18 12:33 AM, vito jeng wrote:
> > >>> Matthias,
> > >>>
> > >>> Thanks for the review.
> > >>> I reply separately in the following sections.
> > >>>
> > >>>
> > >>> ---
> > >>> Vito
> > >>>
> > >>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> matthias@confluent.io
> > >
> > >>> wrote:
> > >>>
> > >>>> Thanks for updating the KIP and sorry for the long pause...
> > >>>>
> > >>>> Seems you did a very thorough investigation of the code. It's useful
> > to
> > >>>> understand what user facing interfaces are affected.
> > >>>
> > >>> (Some parts might be even too detailed for a KIP.)
> > >>>>
> > >>>
> > >>> I also think too detailed. Especially the section `Changes in call
> > >> trace`.
> > >>> Do you think it should be removed?
> > >>>
> > >>>
> > >>>>
> > >>>> To summarize my current understanding of your KIP, the main change
> is
> > to
> > >>>> introduce new exceptions that extend `InvalidStateStoreException`.
> > >>>>
> > >>>
> > >>> yep. Keep compatibility in this KIP is important things.
> > >>> I think the best way is that all new exceptions extend from
> > >>> `InvalidStateStoreException`.
> > >>>
> > >>>
> > >>>>
> > >>>> Some questions:
> > >>>>
> > >>>> - Why do we need ```? Could `InvalidStateStoreException` be used
> for
> > >>>> this purpose?
> > >>>>
> > >>>
> > >>> Does this question miss some word?
> > >>>
> > >>>
> > >>>>
> > >>>> - What the superclass of `StateStoreStreamThreadNotRunni
> ngException`
> > >>>> is? Should it be `InvalidStateStoreException` or
> > >> `StateStoreFailException`
> > >>>> ?
> > >>>>
> > >>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> > >>>>
> > >>>>
> > >>> I apologize for not well written parts. I tried to modify some code
> in
> > >> the
> > >>> recent period and modify the KIP.
> > >>> The modification is now complete. Please look again.
> > >>>
> > >>>
> > >>>>
> > >>>>
> > >>>> -Matthias
> > >>>>
> > >>>>
> > >>>> On 2/21/18 5:15 PM, vito jeng wrote:
> > >>>>> Matthias,
> > >>>>>
> > >>>>> Sorry for not response these days.
> > >>>>> I just finished it. Please have a look. :)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> ---
> > >>>>> Vito
> > >>>>>
> > >>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> > >> matthias@confluent.io>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Is there any update on this KIP?
> > >>>>>>
> > >>>>>> -Matthias
> > >>>>>>
> > >>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> > >>>>>>> Matthias,
> > >>>>>>>
> > >>>>>>> Thank you for your response.
> > >>>>>>>
> > >>>>>>> I think you are right. We need to look at the state both of
> > >>>>>>> KafkaStreams and StreamThread.
> > >>>>>>>
> > >>>>>>> After further understanding of KafkaStreams thread and state
> store,
> > >>>>>>> I am currently rewriting the KIP.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> ---
> > >>>>>>> Vito
> > >>>>>>>
> > >>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> > >>>> matthias@confluent.io>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Vito,
> > >>>>>>>>
> > >>>>>>>> Sorry for this late reply.
> > >>>>>>>>
> > >>>>>>>> There can be two cases:
> > >>>>>>>>
> > >>>>>>>> - either a store got migrated way and thus, is not hosted an
> the
> > >>>>>>>> application instance anymore,
> > >>>>>>>> - or, a store is hosted but the instance is in state rebalance
> > >>>>>>>>
> > >>>>>>>> For the first case, users need to rediscover the store. For the
> > >> second
> > >>>>>>>> case, they need to wait until rebalance is finished.
> > >>>>>>>>
> > >>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> > NOT_RUNNING,
> > >>>>>>>> uses cannot query at all and thus they cannot rediscover or
> retry.
> > >>>>>>>>
> > >>>>>>>> Does this make sense?
> > >>>>>>>>
> > >>>>>>>> -Matthias
> > >>>>>>>>
> > >>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> > >>>>>>>>> Matthias,
> > >>>>>>>>>
> > >>>>>>>>> I try to clarify some concept.
> > >>>>>>>>>
> > >>>>>>>>> When streams state is REBALANCING, it means the user can just
> > plain
> > >>>>>>>> retry.
> > >>>>>>>>>
> > >>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING,
> > it
> > >>>>>> means
> > >>>>>>>>> state store migrated to another instance, the user needs to
> > >>>> rediscover
> > >>>>>>>> the
> > >>>>>>>>> store.
> > >>>>>>>>>
> > >>>>>>>>> Is my understanding correct?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ---
> > >>>>>>>>> Vito
> > >>>>>>>>>
> > >>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> > >>>>>> matthias@confluent.io>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Thanks for the KIP Vito!
> > >>>>>>>>>>
> > >>>>>>>>>> I agree with what Guozhang said. The original idea of the Jira
> > >> was,
> > >>>> to
> > >>>>>>>>>> give different exceptions for different "recovery" strategies
> to
> > >> the
> > >>>>>>>> user.
> > >>>>>>>>>>
> > >>>>>>>>>> For example, if a store is currently recreated, a user just
> need
> > >> to
> > >>>>>> wait
> > >>>>>>>>>> and can query the store later. On the other hand, if a store
> go
> > >>>>>> migrated
> > >>>>>>>>>> to another instance, a user needs to rediscover the store
> > instead
> > >>>> of a
> > >>>>>>>>>> "plain retry".
> > >>>>>>>>>>
> > >>>>>>>>>> Fatal errors might be a third category.
> > >>>>>>>>>>
> > >>>>>>>>>> Not sure if there is something else?
> > >>>>>>>>>>
> > >>>>>>>>>> Anyway, the KIP should contain a section that talks about this
> > >> ideas
> > >>>>>> and
> > >>>>>>>>>> reasoning.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> -Matthias
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > >>>>>>>>>>> Thanks for writing up the KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first
> is
> > >> what
> > >>>>>>>>>>> categories of errors we want to notify the users, if we only
> > >> wants
> > >>>> to
> > >>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
> rename
> > >> the
> > >>>>>>>>>>> proposed StateStoreMigratedException /
> > StateStoreClosedException
> > >>>>>>>> classes.
> > >>>>>>>>>>> And then from there we should list what are the possible
> > internal
> > >>>>>>>>>>> exceptions ever thrown in those APIs in the call trace, and
> > which
> > >>>>>>>>>>> exceptions should be wrapped to what others, and which ones
> > >> should
> > >>>> be
> > >>>>>>>>>>> handled without re-throwing, and which ones should not be
> > wrapped
> > >>>> at
> > >>>>>>>> all
> > >>>>>>>>>>> but directly thrown to user's face.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Guozhang
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> > vito@is-land.com.tw>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'd like to start discuss KIP-216:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> > >> different+errors
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Please have a look.
> > >>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> Vito
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >
> >
> >
>
--
-- Guozhang
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Hi, John,
Thanks for the comment!
KIP typically discuss the public interfaces, non-user-facing detail is not
necessary.
I think your opinion is correct.
But in this KIP, when a state store exception throw, the user will see the
full exception stacktrace in the log, including the internal exception.
So I think it would be better we made some description about the internal
exception in the KIP.
About the base class InvalidStateStoreException, I agree with Guozhang's
said.
We should expose it as public for people who do not want to handle the
internal exception differently.
Users can have more flexibility in how to handle errors.
Thanks.
---
Vito
On Sat, Apr 21, 2018 at 12:14 AM, John Roesler <jo...@confluent.io> wrote:
> Hi Vito,
>
> Thanks for the KIP!
>
> I think it's much nicer to give callers different exceptions to tell them
> whether the state store got migrated, whether it's still initializing, or
> whether there's some unrecoverable error.
>
> In the KIP, it's typically not necessary to discuss non-user-facing details
> such as what exceptions we will throw internally. The KIP is primarily to
> discuss public interface changes.
>
> You might consider simply removing all the internal details from the KIP,
> which will have the dual advantage that it makes the KIP smaller and easier
> to agree on, as well as giving you more freedom in the internal details
> when it comes to implementation.
>
> I like your decision to have your refined exceptions extend
> InvalidStateStoreException to ensure backward compatibility. Since we want
> to encourage callers to catch the more specific exceptions, and we don't
> expect to ever throw a raw InvalidStateStoreException anymore, you might
> consider adding the @Deprecated annotation to InvalidStateStoreException.
> This will gently encourage callers to migrate to the new exception and open
> the possibility of removing InvalidStateStoreException entirely in a future
> release.
>
> Thanks,
> -John
>
> On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Thanks for clarification! That makes sense to me.
> >
> > Can you update the KIP to make those suggestions explicit?
> >
> >
> > -Matthias
> >
> > On 4/18/18 2:19 PM, vito jeng wrote:
> > > Matthias,
> > >
> > > Thanks for the feedback!
> > >
> > >> It's up to you to keep the details part in the KIP or not.
> > >
> > > Got it!
> > >
> > >> The (incomplete) question was, if we need `StateStoreFailException` or
> > >> if existing `InvalidStateStoreException` could be used? Do you suggest
> > >> that `InvalidStateStoreException` is not thrown at all anymore, but
> only
> > >> the new sub-classes (just to get a better understanding).
> > >
> > > Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> > > anymore,
> > > but only new sub-classes.
> > >
> > >> Not sure what this sentence means:
> > >>> The internal exception will be wrapped as category exception finally.
> > >> Can you elaborate?
> > >
> > > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > > `StreamThreadNotRunningException`(internal exception).
> > > And then the internal exception will be wrapped as
> > > `StateStoreRetryableException` or `StateStoreFailException` during the
> > > `KafkaStreams.store()` and throw to the user.
> > >
> > >
> > >> Can you explain the purpose of the "internal exceptions". It's unclear
> > > to me atm why they are introduced.
> > >
> > > Hmmm...the purpose of the "internal exceptions" is to distinguish
> between
> > > the different kinds of InvalidStateStoreException.
> > > The original idea is that we can distinguish different state store
> > > exception for
> > > different handling.
> > > But to be honest, I am not quite sure this is necessary. Maybe have
> > > some change during implementation.
> > >
> > > Does it make sense?
> > >
> > >
> > >
> > >
> > > ---
> > > Vito
> > >
> > > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <
> matthias@confluent.io>
> > > wrote:
> > >
> > >> Thanks for the update Vito!
> > >>
> > >> It's up to you to keep the details part in the KIP or not.
> > >>
> > >>
> > >> The (incomplete) question was, if we need `StateStoreFailException` or
> > >> if existing `InvalidStateStoreException` could be used? Do you suggest
> > >> that `InvalidStateStoreException` is not thrown at all anymore, but
> only
> > >> the new sub-classes (just to get a better understanding).
> > >>
> > >>
> > >> Not sure what this sentence means:
> > >>
> > >>> The internal exception will be wrapped as category exception finally.
> > >>
> > >> Can you elaborate?
> > >>
> > >>
> > >> Can you explain the purpose of the "internal exceptions". It's unclear
> > >> to me atm why they are introduced.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 4/10/18 12:33 AM, vito jeng wrote:
> > >>> Matthias,
> > >>>
> > >>> Thanks for the review.
> > >>> I reply separately in the following sections.
> > >>>
> > >>>
> > >>> ---
> > >>> Vito
> > >>>
> > >>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <
> matthias@confluent.io
> > >
> > >>> wrote:
> > >>>
> > >>>> Thanks for updating the KIP and sorry for the long pause...
> > >>>>
> > >>>> Seems you did a very thorough investigation of the code. It's useful
> > to
> > >>>> understand what user facing interfaces are affected.
> > >>>
> > >>> (Some parts might be even too detailed for a KIP.)
> > >>>>
> > >>>
> > >>> I also think too detailed. Especially the section `Changes in call
> > >> trace`.
> > >>> Do you think it should be removed?
> > >>>
> > >>>
> > >>>>
> > >>>> To summarize my current understanding of your KIP, the main change
> is
> > to
> > >>>> introduce new exceptions that extend `InvalidStateStoreException`.
> > >>>>
> > >>>
> > >>> yep. Keep compatibility in this KIP is important things.
> > >>> I think the best way is that all new exceptions extend from
> > >>> `InvalidStateStoreException`.
> > >>>
> > >>>
> > >>>>
> > >>>> Some questions:
> > >>>>
> > >>>> - Why do we need ```? Could `InvalidStateStoreException` be used
> for
> > >>>> this purpose?
> > >>>>
> > >>>
> > >>> Does this question miss some word?
> > >>>
> > >>>
> > >>>>
> > >>>> - What the superclass of `StateStoreStreamThreadNotRunni
> ngException`
> > >>>> is? Should it be `InvalidStateStoreException` or
> > >> `StateStoreFailException`
> > >>>> ?
> > >>>>
> > >>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> > >>>>
> > >>>>
> > >>> I apologize for not well written parts. I tried to modify some code
> in
> > >> the
> > >>> recent period and modify the KIP.
> > >>> The modification is now complete. Please look again.
> > >>>
> > >>>
> > >>>>
> > >>>>
> > >>>> -Matthias
> > >>>>
> > >>>>
> > >>>> On 2/21/18 5:15 PM, vito jeng wrote:
> > >>>>> Matthias,
> > >>>>>
> > >>>>> Sorry for not response these days.
> > >>>>> I just finished it. Please have a look. :)
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> ---
> > >>>>> Vito
> > >>>>>
> > >>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> > >> matthias@confluent.io>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Is there any update on this KIP?
> > >>>>>>
> > >>>>>> -Matthias
> > >>>>>>
> > >>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> > >>>>>>> Matthias,
> > >>>>>>>
> > >>>>>>> Thank you for your response.
> > >>>>>>>
> > >>>>>>> I think you are right. We need to look at the state both of
> > >>>>>>> KafkaStreams and StreamThread.
> > >>>>>>>
> > >>>>>>> After further understanding of KafkaStreams thread and state
> store,
> > >>>>>>> I am currently rewriting the KIP.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> ---
> > >>>>>>> Vito
> > >>>>>>>
> > >>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> > >>>> matthias@confluent.io>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Vito,
> > >>>>>>>>
> > >>>>>>>> Sorry for this late reply.
> > >>>>>>>>
> > >>>>>>>> There can be two cases:
> > >>>>>>>>
> > >>>>>>>> - either a store got migrated way and thus, is not hosted an
> the
> > >>>>>>>> application instance anymore,
> > >>>>>>>> - or, a store is hosted but the instance is in state rebalance
> > >>>>>>>>
> > >>>>>>>> For the first case, users need to rediscover the store. For the
> > >> second
> > >>>>>>>> case, they need to wait until rebalance is finished.
> > >>>>>>>>
> > >>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> > NOT_RUNNING,
> > >>>>>>>> uses cannot query at all and thus they cannot rediscover or
> retry.
> > >>>>>>>>
> > >>>>>>>> Does this make sense?
> > >>>>>>>>
> > >>>>>>>> -Matthias
> > >>>>>>>>
> > >>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> > >>>>>>>>> Matthias,
> > >>>>>>>>>
> > >>>>>>>>> I try to clarify some concept.
> > >>>>>>>>>
> > >>>>>>>>> When streams state is REBALANCING, it means the user can just
> > plain
> > >>>>>>>> retry.
> > >>>>>>>>>
> > >>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING,
> > it
> > >>>>>> means
> > >>>>>>>>> state store migrated to another instance, the user needs to
> > >>>> rediscover
> > >>>>>>>> the
> > >>>>>>>>> store.
> > >>>>>>>>>
> > >>>>>>>>> Is my understanding correct?
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> ---
> > >>>>>>>>> Vito
> > >>>>>>>>>
> > >>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> > >>>>>> matthias@confluent.io>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Thanks for the KIP Vito!
> > >>>>>>>>>>
> > >>>>>>>>>> I agree with what Guozhang said. The original idea of the Jira
> > >> was,
> > >>>> to
> > >>>>>>>>>> give different exceptions for different "recovery" strategies
> to
> > >> the
> > >>>>>>>> user.
> > >>>>>>>>>>
> > >>>>>>>>>> For example, if a store is currently recreated, a user just
> need
> > >> to
> > >>>>>> wait
> > >>>>>>>>>> and can query the store later. On the other hand, if a store
> go
> > >>>>>> migrated
> > >>>>>>>>>> to another instance, a user needs to rediscover the store
> > instead
> > >>>> of a
> > >>>>>>>>>> "plain retry".
> > >>>>>>>>>>
> > >>>>>>>>>> Fatal errors might be a third category.
> > >>>>>>>>>>
> > >>>>>>>>>> Not sure if there is something else?
> > >>>>>>>>>>
> > >>>>>>>>>> Anyway, the KIP should contain a section that talks about this
> > >> ideas
> > >>>>>> and
> > >>>>>>>>>> reasoning.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> -Matthias
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > >>>>>>>>>>> Thanks for writing up the KIP.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first
> is
> > >> what
> > >>>>>>>>>>> categories of errors we want to notify the users, if we only
> > >> wants
> > >>>> to
> > >>>>>>>>>>> distinguish fatal v.s. retriable then probably we should
> rename
> > >> the
> > >>>>>>>>>>> proposed StateStoreMigratedException /
> > StateStoreClosedException
> > >>>>>>>> classes.
> > >>>>>>>>>>> And then from there we should list what are the possible
> > internal
> > >>>>>>>>>>> exceptions ever thrown in those APIs in the call trace, and
> > which
> > >>>>>>>>>>> exceptions should be wrapped to what others, and which ones
> > >> should
> > >>>> be
> > >>>>>>>>>>> handled without re-throwing, and which ones should not be
> > wrapped
> > >>>> at
> > >>>>>>>> all
> > >>>>>>>>>>> but directly thrown to user's face.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Guozhang
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> > vito@is-land.com.tw>
> > >>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I'd like to start discuss KIP-216:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> > >> different+errors
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Please have a look.
> > >>>>>>>>>>>> Thanks!
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> ---
> > >>>>>>>>>>>> Vito
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >
> >
> >
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by John Roesler <jo...@confluent.io>.
Hi Vito,
Thanks for the KIP!
I think it's much nicer to give callers different exceptions to tell them
whether the state store got migrated, whether it's still initializing, or
whether there's some unrecoverable error.
In the KIP, it's typically not necessary to discuss non-user-facing details
such as what exceptions we will throw internally. The KIP is primarily to
discuss public interface changes.
You might consider simply removing all the internal details from the KIP,
which will have the dual advantage that it makes the KIP smaller and easier
to agree on, as well as giving you more freedom in the internal details
when it comes to implementation.
I like your decision to have your refined exceptions extend
InvalidStateStoreException to ensure backward compatibility. Since we want
to encourage callers to catch the more specific exceptions, and we don't
expect to ever throw a raw InvalidStateStoreException anymore, you might
consider adding the @Deprecated annotation to InvalidStateStoreException.
This will gently encourage callers to migrate to the new exception and open
the possibility of removing InvalidStateStoreException entirely in a future
release.
Thanks,
-John
On Thu, Apr 19, 2018 at 8:58 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for clarification! That makes sense to me.
>
> Can you update the KIP to make those suggestions explicit?
>
>
> -Matthias
>
> On 4/18/18 2:19 PM, vito jeng wrote:
> > Matthias,
> >
> > Thanks for the feedback!
> >
> >> It's up to you to keep the details part in the KIP or not.
> >
> > Got it!
> >
> >> The (incomplete) question was, if we need `StateStoreFailException` or
> >> if existing `InvalidStateStoreException` could be used? Do you suggest
> >> that `InvalidStateStoreException` is not thrown at all anymore, but only
> >> the new sub-classes (just to get a better understanding).
> >
> > Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> > anymore,
> > but only new sub-classes.
> >
> >> Not sure what this sentence means:
> >>> The internal exception will be wrapped as category exception finally.
> >> Can you elaborate?
> >
> > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > `StreamThreadNotRunningException`(internal exception).
> > And then the internal exception will be wrapped as
> > `StateStoreRetryableException` or `StateStoreFailException` during the
> > `KafkaStreams.store()` and throw to the user.
> >
> >
> >> Can you explain the purpose of the "internal exceptions". It's unclear
> > to me atm why they are introduced.
> >
> > Hmmm...the purpose of the "internal exceptions" is to distinguish between
> > the different kinds of InvalidStateStoreException.
> > The original idea is that we can distinguish different state store
> > exception for
> > different handling.
> > But to be honest, I am not quite sure this is necessary. Maybe have
> > some change during implementation.
> >
> > Does it make sense?
> >
> >
> >
> >
> > ---
> > Vito
> >
> > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Thanks for the update Vito!
> >>
> >> It's up to you to keep the details part in the KIP or not.
> >>
> >>
> >> The (incomplete) question was, if we need `StateStoreFailException` or
> >> if existing `InvalidStateStoreException` could be used? Do you suggest
> >> that `InvalidStateStoreException` is not thrown at all anymore, but only
> >> the new sub-classes (just to get a better understanding).
> >>
> >>
> >> Not sure what this sentence means:
> >>
> >>> The internal exception will be wrapped as category exception finally.
> >>
> >> Can you elaborate?
> >>
> >>
> >> Can you explain the purpose of the "internal exceptions". It's unclear
> >> to me atm why they are introduced.
> >>
> >>
> >> -Matthias
> >>
> >> On 4/10/18 12:33 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Thanks for the review.
> >>> I reply separately in the following sections.
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <matthias@confluent.io
> >
> >>> wrote:
> >>>
> >>>> Thanks for updating the KIP and sorry for the long pause...
> >>>>
> >>>> Seems you did a very thorough investigation of the code. It's useful
> to
> >>>> understand what user facing interfaces are affected.
> >>>
> >>> (Some parts might be even too detailed for a KIP.)
> >>>>
> >>>
> >>> I also think too detailed. Especially the section `Changes in call
> >> trace`.
> >>> Do you think it should be removed?
> >>>
> >>>
> >>>>
> >>>> To summarize my current understanding of your KIP, the main change is
> to
> >>>> introduce new exceptions that extend `InvalidStateStoreException`.
> >>>>
> >>>
> >>> yep. Keep compatibility in this KIP is important things.
> >>> I think the best way is that all new exceptions extend from
> >>> `InvalidStateStoreException`.
> >>>
> >>>
> >>>>
> >>>> Some questions:
> >>>>
> >>>> - Why do we need ```? Could `InvalidStateStoreException` be used for
> >>>> this purpose?
> >>>>
> >>>
> >>> Does this question miss some word?
> >>>
> >>>
> >>>>
> >>>> - What the superclass of `StateStoreStreamThreadNotRunningException`
> >>>> is? Should it be `InvalidStateStoreException` or
> >> `StateStoreFailException`
> >>>> ?
> >>>>
> >>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> >>>>
> >>>>
> >>> I apologize for not well written parts. I tried to modify some code in
> >> the
> >>> recent period and modify the KIP.
> >>> The modification is now complete. Please look again.
> >>>
> >>>
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>> Matthias,
> >>>>>
> >>>>> Sorry for not response these days.
> >>>>> I just finished it. Please have a look. :)
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >> matthias@confluent.io>
> >>>>> wrote:
> >>>>>
> >>>>>> Is there any update on this KIP?
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>> Matthias,
> >>>>>>>
> >>>>>>> Thank you for your response.
> >>>>>>>
> >>>>>>> I think you are right. We need to look at the state both of
> >>>>>>> KafkaStreams and StreamThread.
> >>>>>>>
> >>>>>>> After further understanding of KafkaStreams thread and state store,
> >>>>>>> I am currently rewriting the KIP.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ---
> >>>>>>> Vito
> >>>>>>>
> >>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >>>> matthias@confluent.io>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Vito,
> >>>>>>>>
> >>>>>>>> Sorry for this late reply.
> >>>>>>>>
> >>>>>>>> There can be two cases:
> >>>>>>>>
> >>>>>>>> - either a store got migrated way and thus, is not hosted an the
> >>>>>>>> application instance anymore,
> >>>>>>>> - or, a store is hosted but the instance is in state rebalance
> >>>>>>>>
> >>>>>>>> For the first case, users need to rediscover the store. For the
> >> second
> >>>>>>>> case, they need to wait until rebalance is finished.
> >>>>>>>>
> >>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> NOT_RUNNING,
> >>>>>>>> uses cannot query at all and thus they cannot rediscover or retry.
> >>>>>>>>
> >>>>>>>> Does this make sense?
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>> Matthias,
> >>>>>>>>>
> >>>>>>>>> I try to clarify some concept.
> >>>>>>>>>
> >>>>>>>>> When streams state is REBALANCING, it means the user can just
> plain
> >>>>>>>> retry.
> >>>>>>>>>
> >>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING,
> it
> >>>>>> means
> >>>>>>>>> state store migrated to another instance, the user needs to
> >>>> rediscover
> >>>>>>>> the
> >>>>>>>>> store.
> >>>>>>>>>
> >>>>>>>>> Is my understanding correct?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>> Vito
> >>>>>>>>>
> >>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>>>> matthias@confluent.io>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>
> >>>>>>>>>> I agree with what Guozhang said. The original idea of the Jira
> >> was,
> >>>> to
> >>>>>>>>>> give different exceptions for different "recovery" strategies to
> >> the
> >>>>>>>> user.
> >>>>>>>>>>
> >>>>>>>>>> For example, if a store is currently recreated, a user just need
> >> to
> >>>>>> wait
> >>>>>>>>>> and can query the store later. On the other hand, if a store go
> >>>>>> migrated
> >>>>>>>>>> to another instance, a user needs to rediscover the store
> instead
> >>>> of a
> >>>>>>>>>> "plain retry".
> >>>>>>>>>>
> >>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>
> >>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>
> >>>>>>>>>> Anyway, the KIP should contain a section that talks about this
> >> ideas
> >>>>>> and
> >>>>>>>>>> reasoning.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first is
> >> what
> >>>>>>>>>>> categories of errors we want to notify the users, if we only
> >> wants
> >>>> to
> >>>>>>>>>>> distinguish fatal v.s. retriable then probably we should rename
> >> the
> >>>>>>>>>>> proposed StateStoreMigratedException /
> StateStoreClosedException
> >>>>>>>> classes.
> >>>>>>>>>>> And then from there we should list what are the possible
> internal
> >>>>>>>>>>> exceptions ever thrown in those APIs in the call trace, and
> which
> >>>>>>>>>>> exceptions should be wrapped to what others, and which ones
> >> should
> >>>> be
> >>>>>>>>>>> handled without re-throwing, and which ones should not be
> wrapped
> >>>> at
> >>>>>>>> all
> >>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>
> >>>>>>>>>>> Guozhang
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> vito@is-land.com.tw>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>
> >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> >> different+errors
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Hi, Matthias,
Thanks for the review!
I have updated the KIP, please look again.
Thanks.
---
Vito
On Thu, Apr 19, 2018 at 9:58 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for clarification! That makes sense to me.
>
> Can you update the KIP to make those suggestions explicit?
>
>
> -Matthias
>
> On 4/18/18 2:19 PM, vito jeng wrote:
> > Matthias,
> >
> > Thanks for the feedback!
> >
> >> It's up to you to keep the details part in the KIP or not.
> >
> > Got it!
> >
> >> The (incomplete) question was, if we need `StateStoreFailException` or
> >> if existing `InvalidStateStoreException` could be used? Do you suggest
> >> that `InvalidStateStoreException` is not thrown at all anymore, but only
> >> the new sub-classes (just to get a better understanding).
> >
> > Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> > anymore,
> > but only new sub-classes.
> >
> >> Not sure what this sentence means:
> >>> The internal exception will be wrapped as category exception finally.
> >> Can you elaborate?
> >
> > For example, `StreamThreadStateStoreProvider#stores()` will throw
> > `StreamThreadNotRunningException`(internal exception).
> > And then the internal exception will be wrapped as
> > `StateStoreRetryableException` or `StateStoreFailException` during the
> > `KafkaStreams.store()` and throw to the user.
> >
> >
> >> Can you explain the purpose of the "internal exceptions". It's unclear
> > to me atm why they are introduced.
> >
> > Hmmm...the purpose of the "internal exceptions" is to distinguish between
> > the different kinds of InvalidStateStoreException.
> > The original idea is that we can distinguish different state store
> > exception for
> > different handling.
> > But to be honest, I am not quite sure this is necessary. Maybe have
> > some change during implementation.
> >
> > Does it make sense?
> >
> >
> >
> >
> > ---
> > Vito
> >
> > On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Thanks for the update Vito!
> >>
> >> It's up to you to keep the details part in the KIP or not.
> >>
> >>
> >> The (incomplete) question was, if we need `StateStoreFailException` or
> >> if existing `InvalidStateStoreException` could be used? Do you suggest
> >> that `InvalidStateStoreException` is not thrown at all anymore, but only
> >> the new sub-classes (just to get a better understanding).
> >>
> >>
> >> Not sure what this sentence means:
> >>
> >>> The internal exception will be wrapped as category exception finally.
> >>
> >> Can you elaborate?
> >>
> >>
> >> Can you explain the purpose of the "internal exceptions". It's unclear
> >> to me atm why they are introduced.
> >>
> >>
> >> -Matthias
> >>
> >> On 4/10/18 12:33 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Thanks for the review.
> >>> I reply separately in the following sections.
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <matthias@confluent.io
> >
> >>> wrote:
> >>>
> >>>> Thanks for updating the KIP and sorry for the long pause...
> >>>>
> >>>> Seems you did a very thorough investigation of the code. It's useful
> to
> >>>> understand what user facing interfaces are affected.
> >>>
> >>> (Some parts might be even too detailed for a KIP.)
> >>>>
> >>>
> >>> I also think too detailed. Especially the section `Changes in call
> >> trace`.
> >>> Do you think it should be removed?
> >>>
> >>>
> >>>>
> >>>> To summarize my current understanding of your KIP, the main change is
> to
> >>>> introduce new exceptions that extend `InvalidStateStoreException`.
> >>>>
> >>>
> >>> yep. Keep compatibility in this KIP is important things.
> >>> I think the best way is that all new exceptions extend from
> >>> `InvalidStateStoreException`.
> >>>
> >>>
> >>>>
> >>>> Some questions:
> >>>>
> >>>> - Why do we need ```? Could `InvalidStateStoreException` be used for
> >>>> this purpose?
> >>>>
> >>>
> >>> Does this question miss some word?
> >>>
> >>>
> >>>>
> >>>> - What the superclass of `StateStoreStreamThreadNotRunningException`
> >>>> is? Should it be `InvalidStateStoreException` or
> >> `StateStoreFailException`
> >>>> ?
> >>>>
> >>>> - Is `StateStoreClosed` a fatal or retryable exception ?
> >>>>
> >>>>
> >>> I apologize for not well written parts. I tried to modify some code in
> >> the
> >>> recent period and modify the KIP.
> >>> The modification is now complete. Please look again.
> >>>
> >>>
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 2/21/18 5:15 PM, vito jeng wrote:
> >>>>> Matthias,
> >>>>>
> >>>>> Sorry for not response these days.
> >>>>> I just finished it. Please have a look. :)
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> >> matthias@confluent.io>
> >>>>> wrote:
> >>>>>
> >>>>>> Is there any update on this KIP?
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>>>> Matthias,
> >>>>>>>
> >>>>>>> Thank you for your response.
> >>>>>>>
> >>>>>>> I think you are right. We need to look at the state both of
> >>>>>>> KafkaStreams and StreamThread.
> >>>>>>>
> >>>>>>> After further understanding of KafkaStreams thread and state store,
> >>>>>>> I am currently rewriting the KIP.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ---
> >>>>>>> Vito
> >>>>>>>
> >>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >>>> matthias@confluent.io>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Vito,
> >>>>>>>>
> >>>>>>>> Sorry for this late reply.
> >>>>>>>>
> >>>>>>>> There can be two cases:
> >>>>>>>>
> >>>>>>>> - either a store got migrated way and thus, is not hosted an the
> >>>>>>>> application instance anymore,
> >>>>>>>> - or, a store is hosted but the instance is in state rebalance
> >>>>>>>>
> >>>>>>>> For the first case, users need to rediscover the store. For the
> >> second
> >>>>>>>> case, they need to wait until rebalance is finished.
> >>>>>>>>
> >>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or
> NOT_RUNNING,
> >>>>>>>> uses cannot query at all and thus they cannot rediscover or retry.
> >>>>>>>>
> >>>>>>>> Does this make sense?
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>>>> Matthias,
> >>>>>>>>>
> >>>>>>>>> I try to clarify some concept.
> >>>>>>>>>
> >>>>>>>>> When streams state is REBALANCING, it means the user can just
> plain
> >>>>>>>> retry.
> >>>>>>>>>
> >>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING,
> it
> >>>>>> means
> >>>>>>>>> state store migrated to another instance, the user needs to
> >>>> rediscover
> >>>>>>>> the
> >>>>>>>>> store.
> >>>>>>>>>
> >>>>>>>>> Is my understanding correct?
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>> Vito
> >>>>>>>>>
> >>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>>>> matthias@confluent.io>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>>>
> >>>>>>>>>> I agree with what Guozhang said. The original idea of the Jira
> >> was,
> >>>> to
> >>>>>>>>>> give different exceptions for different "recovery" strategies to
> >> the
> >>>>>>>> user.
> >>>>>>>>>>
> >>>>>>>>>> For example, if a store is currently recreated, a user just need
> >> to
> >>>>>> wait
> >>>>>>>>>> and can query the store later. On the other hand, if a store go
> >>>>>> migrated
> >>>>>>>>>> to another instance, a user needs to rediscover the store
> instead
> >>>> of a
> >>>>>>>>>> "plain retry".
> >>>>>>>>>>
> >>>>>>>>>> Fatal errors might be a third category.
> >>>>>>>>>>
> >>>>>>>>>> Not sure if there is something else?
> >>>>>>>>>>
> >>>>>>>>>> Anyway, the KIP should contain a section that talks about this
> >> ideas
> >>>>>> and
> >>>>>>>>>> reasoning.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>>>
> >>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first is
> >> what
> >>>>>>>>>>> categories of errors we want to notify the users, if we only
> >> wants
> >>>> to
> >>>>>>>>>>> distinguish fatal v.s. retriable then probably we should rename
> >> the
> >>>>>>>>>>> proposed StateStoreMigratedException /
> StateStoreClosedException
> >>>>>>>> classes.
> >>>>>>>>>>> And then from there we should list what are the possible
> internal
> >>>>>>>>>>> exceptions ever thrown in those APIs in the call trace, and
> which
> >>>>>>>>>>> exceptions should be wrapped to what others, and which ones
> >> should
> >>>> be
> >>>>>>>>>>> handled without re-throwing, and which ones should not be
> wrapped
> >>>> at
> >>>>>>>> all
> >>>>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>>>
> >>>>>>>>>>> Guozhang
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <
> vito@is-land.com.tw>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>>>
> >>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> >> different+errors
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please have a look.
> >>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---
> >>>>>>>>>>>> Vito
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks for clarification! That makes sense to me.
Can you update the KIP to make those suggestions explicit?
-Matthias
On 4/18/18 2:19 PM, vito jeng wrote:
> Matthias,
>
> Thanks for the feedback!
>
>> It's up to you to keep the details part in the KIP or not.
>
> Got it!
>
>> The (incomplete) question was, if we need `StateStoreFailException` or
>> if existing `InvalidStateStoreException` could be used? Do you suggest
>> that `InvalidStateStoreException` is not thrown at all anymore, but only
>> the new sub-classes (just to get a better understanding).
>
> Yes. I suggest that `InvalidStateStoreException` is not thrown at all
> anymore,
> but only new sub-classes.
>
>> Not sure what this sentence means:
>>> The internal exception will be wrapped as category exception finally.
>> Can you elaborate?
>
> For example, `StreamThreadStateStoreProvider#stores()` will throw
> `StreamThreadNotRunningException`(internal exception).
> And then the internal exception will be wrapped as
> `StateStoreRetryableException` or `StateStoreFailException` during the
> `KafkaStreams.store()` and throw to the user.
>
>
>> Can you explain the purpose of the "internal exceptions". It's unclear
> to me atm why they are introduced.
>
> Hmmm...the purpose of the "internal exceptions" is to distinguish between
> the different kinds of InvalidStateStoreException.
> The original idea is that we can distinguish different state store
> exception for
> different handling.
> But to be honest, I am not quite sure this is necessary. Maybe have
> some change during implementation.
>
> Does it make sense?
>
>
>
>
> ---
> Vito
>
> On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Thanks for the update Vito!
>>
>> It's up to you to keep the details part in the KIP or not.
>>
>>
>> The (incomplete) question was, if we need `StateStoreFailException` or
>> if existing `InvalidStateStoreException` could be used? Do you suggest
>> that `InvalidStateStoreException` is not thrown at all anymore, but only
>> the new sub-classes (just to get a better understanding).
>>
>>
>> Not sure what this sentence means:
>>
>>> The internal exception will be wrapped as category exception finally.
>>
>> Can you elaborate?
>>
>>
>> Can you explain the purpose of the "internal exceptions". It's unclear
>> to me atm why they are introduced.
>>
>>
>> -Matthias
>>
>> On 4/10/18 12:33 AM, vito jeng wrote:
>>> Matthias,
>>>
>>> Thanks for the review.
>>> I reply separately in the following sections.
>>>
>>>
>>> ---
>>> Vito
>>>
>>> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Thanks for updating the KIP and sorry for the long pause...
>>>>
>>>> Seems you did a very thorough investigation of the code. It's useful to
>>>> understand what user facing interfaces are affected.
>>>
>>> (Some parts might be even too detailed for a KIP.)
>>>>
>>>
>>> I also think too detailed. Especially the section `Changes in call
>> trace`.
>>> Do you think it should be removed?
>>>
>>>
>>>>
>>>> To summarize my current understanding of your KIP, the main change is to
>>>> introduce new exceptions that extend `InvalidStateStoreException`.
>>>>
>>>
>>> yep. Keep compatibility in this KIP is important things.
>>> I think the best way is that all new exceptions extend from
>>> `InvalidStateStoreException`.
>>>
>>>
>>>>
>>>> Some questions:
>>>>
>>>> - Why do we need ```? Could `InvalidStateStoreException` be used for
>>>> this purpose?
>>>>
>>>
>>> Does this question miss some word?
>>>
>>>
>>>>
>>>> - What the superclass of `StateStoreStreamThreadNotRunningException`
>>>> is? Should it be `InvalidStateStoreException` or
>> `StateStoreFailException`
>>>> ?
>>>>
>>>> - Is `StateStoreClosed` a fatal or retryable exception ?
>>>>
>>>>
>>> I apologize for not well written parts. I tried to modify some code in
>> the
>>> recent period and modify the KIP.
>>> The modification is now complete. Please look again.
>>>
>>>
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 2/21/18 5:15 PM, vito jeng wrote:
>>>>> Matthias,
>>>>>
>>>>> Sorry for not response these days.
>>>>> I just finished it. Please have a look. :)
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
>> matthias@confluent.io>
>>>>> wrote:
>>>>>
>>>>>> Is there any update on this KIP?
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>>>> Matthias,
>>>>>>>
>>>>>>> Thank you for your response.
>>>>>>>
>>>>>>> I think you are right. We need to look at the state both of
>>>>>>> KafkaStreams and StreamThread.
>>>>>>>
>>>>>>> After further understanding of KafkaStreams thread and state store,
>>>>>>> I am currently rewriting the KIP.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Vito
>>>>>>>
>>>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>>>> matthias@confluent.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Vito,
>>>>>>>>
>>>>>>>> Sorry for this late reply.
>>>>>>>>
>>>>>>>> There can be two cases:
>>>>>>>>
>>>>>>>> - either a store got migrated way and thus, is not hosted an the
>>>>>>>> application instance anymore,
>>>>>>>> - or, a store is hosted but the instance is in state rebalance
>>>>>>>>
>>>>>>>> For the first case, users need to rediscover the store. For the
>> second
>>>>>>>> case, they need to wait until rebalance is finished.
>>>>>>>>
>>>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
>>>>>>>> uses cannot query at all and thus they cannot rediscover or retry.
>>>>>>>>
>>>>>>>> Does this make sense?
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>>>> Matthias,
>>>>>>>>>
>>>>>>>>> I try to clarify some concept.
>>>>>>>>>
>>>>>>>>> When streams state is REBALANCING, it means the user can just plain
>>>>>>>> retry.
>>>>>>>>>
>>>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
>>>>>> means
>>>>>>>>> state store migrated to another instance, the user needs to
>>>> rediscover
>>>>>>>> the
>>>>>>>>> store.
>>>>>>>>>
>>>>>>>>> Is my understanding correct?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Vito
>>>>>>>>>
>>>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>>>>>> matthias@confluent.io>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>>>
>>>>>>>>>> I agree with what Guozhang said. The original idea of the Jira
>> was,
>>>> to
>>>>>>>>>> give different exceptions for different "recovery" strategies to
>> the
>>>>>>>> user.
>>>>>>>>>>
>>>>>>>>>> For example, if a store is currently recreated, a user just need
>> to
>>>>>> wait
>>>>>>>>>> and can query the store later. On the other hand, if a store go
>>>>>> migrated
>>>>>>>>>> to another instance, a user needs to rediscover the store instead
>>>> of a
>>>>>>>>>> "plain retry".
>>>>>>>>>>
>>>>>>>>>> Fatal errors might be a third category.
>>>>>>>>>>
>>>>>>>>>> Not sure if there is something else?
>>>>>>>>>>
>>>>>>>>>> Anyway, the KIP should contain a section that talks about this
>> ideas
>>>>>> and
>>>>>>>>>> reasoning.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Matthias
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>>>
>>>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first is
>> what
>>>>>>>>>>> categories of errors we want to notify the users, if we only
>> wants
>>>> to
>>>>>>>>>>> distinguish fatal v.s. retriable then probably we should rename
>> the
>>>>>>>>>>> proposed StateStoreMigratedException / StateStoreClosedException
>>>>>>>> classes.
>>>>>>>>>>> And then from there we should list what are the possible internal
>>>>>>>>>>> exceptions ever thrown in those APIs in the call trace, and which
>>>>>>>>>>> exceptions should be wrapped to what others, and which ones
>> should
>>>> be
>>>>>>>>>>> handled without re-throwing, and which ones should not be wrapped
>>>> at
>>>>>>>> all
>>>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>>>
>>>>>>>>>>> Guozhang
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>>>
>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
>> different+errors
>>>>>>>>>>>>
>>>>>>>>>>>> Please have a look.
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> ---
>>>>>>>>>>>> Vito
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
Thanks for the feedback!
> It's up to you to keep the details part in the KIP or not.
Got it!
> The (incomplete) question was, if we need `StateStoreFailException` or
> if existing `InvalidStateStoreException` could be used? Do you suggest
> that `InvalidStateStoreException` is not thrown at all anymore, but only
> the new sub-classes (just to get a better understanding).
Yes. I suggest that `InvalidStateStoreException` is not thrown at all
anymore,
but only new sub-classes.
> Not sure what this sentence means:
>> The internal exception will be wrapped as category exception finally.
> Can you elaborate?
For example, `StreamThreadStateStoreProvider#stores()` will throw
`StreamThreadNotRunningException`(internal exception).
And then the internal exception will be wrapped as
`StateStoreRetryableException` or `StateStoreFailException` during the
`KafkaStreams.store()` and throw to the user.
> Can you explain the purpose of the "internal exceptions". It's unclear
to me atm why they are introduced.
Hmmm...the purpose of the "internal exceptions" is to distinguish between
the different kinds of InvalidStateStoreException.
The original idea is that we can distinguish different state store
exception for
different handling.
But to be honest, I am not quite sure this is necessary. Maybe have
some change during implementation.
Does it make sense?
---
Vito
On Mon, Apr 16, 2018 at 5:59 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for the update Vito!
>
> It's up to you to keep the details part in the KIP or not.
>
>
> The (incomplete) question was, if we need `StateStoreFailException` or
> if existing `InvalidStateStoreException` could be used? Do you suggest
> that `InvalidStateStoreException` is not thrown at all anymore, but only
> the new sub-classes (just to get a better understanding).
>
>
> Not sure what this sentence means:
>
> > The internal exception will be wrapped as category exception finally.
>
> Can you elaborate?
>
>
> Can you explain the purpose of the "internal exceptions". It's unclear
> to me atm why they are introduced.
>
>
> -Matthias
>
> On 4/10/18 12:33 AM, vito jeng wrote:
> > Matthias,
> >
> > Thanks for the review.
> > I reply separately in the following sections.
> >
> >
> > ---
> > Vito
> >
> > On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Thanks for updating the KIP and sorry for the long pause...
> >>
> >> Seems you did a very thorough investigation of the code. It's useful to
> >> understand what user facing interfaces are affected.
> >
> > (Some parts might be even too detailed for a KIP.)
> >>
> >
> > I also think too detailed. Especially the section `Changes in call
> trace`.
> > Do you think it should be removed?
> >
> >
> >>
> >> To summarize my current understanding of your KIP, the main change is to
> >> introduce new exceptions that extend `InvalidStateStoreException`.
> >>
> >
> > yep. Keep compatibility in this KIP is important things.
> > I think the best way is that all new exceptions extend from
> > `InvalidStateStoreException`.
> >
> >
> >>
> >> Some questions:
> >>
> >> - Why do we need ```? Could `InvalidStateStoreException` be used for
> >> this purpose?
> >>
> >
> > Does this question miss some word?
> >
> >
> >>
> >> - What the superclass of `StateStoreStreamThreadNotRunningException`
> >> is? Should it be `InvalidStateStoreException` or
> `StateStoreFailException`
> >> ?
> >>
> >> - Is `StateStoreClosed` a fatal or retryable exception ?
> >>
> >>
> > I apologize for not well written parts. I tried to modify some code in
> the
> > recent period and modify the KIP.
> > The modification is now complete. Please look again.
> >
> >
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 2/21/18 5:15 PM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Sorry for not response these days.
> >>> I just finished it. Please have a look. :)
> >>>
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <
> matthias@confluent.io>
> >>> wrote:
> >>>
> >>>> Is there any update on this KIP?
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 1/3/18 12:59 AM, vito jeng wrote:
> >>>>> Matthias,
> >>>>>
> >>>>> Thank you for your response.
> >>>>>
> >>>>> I think you are right. We need to look at the state both of
> >>>>> KafkaStreams and StreamThread.
> >>>>>
> >>>>> After further understanding of KafkaStreams thread and state store,
> >>>>> I am currently rewriting the KIP.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> >> matthias@confluent.io>
> >>>>> wrote:
> >>>>>
> >>>>>> Vito,
> >>>>>>
> >>>>>> Sorry for this late reply.
> >>>>>>
> >>>>>> There can be two cases:
> >>>>>>
> >>>>>> - either a store got migrated way and thus, is not hosted an the
> >>>>>> application instance anymore,
> >>>>>> - or, a store is hosted but the instance is in state rebalance
> >>>>>>
> >>>>>> For the first case, users need to rediscover the store. For the
> second
> >>>>>> case, they need to wait until rebalance is finished.
> >>>>>>
> >>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
> >>>>>> uses cannot query at all and thus they cannot rediscover or retry.
> >>>>>>
> >>>>>> Does this make sense?
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>>>> Matthias,
> >>>>>>>
> >>>>>>> I try to clarify some concept.
> >>>>>>>
> >>>>>>> When streams state is REBALANCING, it means the user can just plain
> >>>>>> retry.
> >>>>>>>
> >>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
> >>>> means
> >>>>>>> state store migrated to another instance, the user needs to
> >> rediscover
> >>>>>> the
> >>>>>>> store.
> >>>>>>>
> >>>>>>> Is my understanding correct?
> >>>>>>>
> >>>>>>>
> >>>>>>> ---
> >>>>>>> Vito
> >>>>>>>
> >>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >>>> matthias@confluent.io>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks for the KIP Vito!
> >>>>>>>>
> >>>>>>>> I agree with what Guozhang said. The original idea of the Jira
> was,
> >> to
> >>>>>>>> give different exceptions for different "recovery" strategies to
> the
> >>>>>> user.
> >>>>>>>>
> >>>>>>>> For example, if a store is currently recreated, a user just need
> to
> >>>> wait
> >>>>>>>> and can query the store later. On the other hand, if a store go
> >>>> migrated
> >>>>>>>> to another instance, a user needs to rediscover the store instead
> >> of a
> >>>>>>>> "plain retry".
> >>>>>>>>
> >>>>>>>> Fatal errors might be a third category.
> >>>>>>>>
> >>>>>>>> Not sure if there is something else?
> >>>>>>>>
> >>>>>>>> Anyway, the KIP should contain a section that talks about this
> ideas
> >>>> and
> >>>>>>>> reasoning.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -Matthias
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>>>> Thanks for writing up the KIP.
> >>>>>>>>>
> >>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first is
> what
> >>>>>>>>> categories of errors we want to notify the users, if we only
> wants
> >> to
> >>>>>>>>> distinguish fatal v.s. retriable then probably we should rename
> the
> >>>>>>>>> proposed StateStoreMigratedException / StateStoreClosedException
> >>>>>> classes.
> >>>>>>>>> And then from there we should list what are the possible internal
> >>>>>>>>> exceptions ever thrown in those APIs in the call trace, and which
> >>>>>>>>> exceptions should be wrapped to what others, and which ones
> should
> >> be
> >>>>>>>>> handled without re-throwing, and which ones should not be wrapped
> >> at
> >>>>>> all
> >>>>>>>>> but directly thrown to user's face.
> >>>>>>>>>
> >>>>>>>>> Guozhang
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>>>
> >>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+
> different+errors
> >>>>>>>>>>
> >>>>>>>>>> Please have a look.
> >>>>>>>>>> Thanks!
> >>>>>>>>>>
> >>>>>>>>>> ---
> >>>>>>>>>> Vito
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks for the update Vito!
It's up to you to keep the details part in the KIP or not.
The (incomplete) question was, if we need `StateStoreFailException` or
if existing `InvalidStateStoreException` could be used? Do you suggest
that `InvalidStateStoreException` is not thrown at all anymore, but only
the new sub-classes (just to get a better understanding).
Not sure what this sentence means:
> The internal exception will be wrapped as category exception finally.
Can you elaborate?
Can you explain the purpose of the "internal exceptions". It's unclear
to me atm why they are introduced.
-Matthias
On 4/10/18 12:33 AM, vito jeng wrote:
> Matthias,
>
> Thanks for the review.
> I reply separately in the following sections.
>
>
> ---
> Vito
>
> On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Thanks for updating the KIP and sorry for the long pause...
>>
>> Seems you did a very thorough investigation of the code. It's useful to
>> understand what user facing interfaces are affected.
>
> (Some parts might be even too detailed for a KIP.)
>>
>
> I also think too detailed. Especially the section `Changes in call trace`.
> Do you think it should be removed?
>
>
>>
>> To summarize my current understanding of your KIP, the main change is to
>> introduce new exceptions that extend `InvalidStateStoreException`.
>>
>
> yep. Keep compatibility in this KIP is important things.
> I think the best way is that all new exceptions extend from
> `InvalidStateStoreException`.
>
>
>>
>> Some questions:
>>
>> - Why do we need ```? Could `InvalidStateStoreException` be used for
>> this purpose?
>>
>
> Does this question miss some word?
>
>
>>
>> - What the superclass of `StateStoreStreamThreadNotRunningException`
>> is? Should it be `InvalidStateStoreException` or `StateStoreFailException`
>> ?
>>
>> - Is `StateStoreClosed` a fatal or retryable exception ?
>>
>>
> I apologize for not well written parts. I tried to modify some code in the
> recent period and modify the KIP.
> The modification is now complete. Please look again.
>
>
>>
>>
>> -Matthias
>>
>>
>> On 2/21/18 5:15 PM, vito jeng wrote:
>>> Matthias,
>>>
>>> Sorry for not response these days.
>>> I just finished it. Please have a look. :)
>>>
>>>
>>>
>>> ---
>>> Vito
>>>
>>> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Is there any update on this KIP?
>>>>
>>>> -Matthias
>>>>
>>>> On 1/3/18 12:59 AM, vito jeng wrote:
>>>>> Matthias,
>>>>>
>>>>> Thank you for your response.
>>>>>
>>>>> I think you are right. We need to look at the state both of
>>>>> KafkaStreams and StreamThread.
>>>>>
>>>>> After further understanding of KafkaStreams thread and state store,
>>>>> I am currently rewriting the KIP.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
>> matthias@confluent.io>
>>>>> wrote:
>>>>>
>>>>>> Vito,
>>>>>>
>>>>>> Sorry for this late reply.
>>>>>>
>>>>>> There can be two cases:
>>>>>>
>>>>>> - either a store got migrated way and thus, is not hosted an the
>>>>>> application instance anymore,
>>>>>> - or, a store is hosted but the instance is in state rebalance
>>>>>>
>>>>>> For the first case, users need to rediscover the store. For the second
>>>>>> case, they need to wait until rebalance is finished.
>>>>>>
>>>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
>>>>>> uses cannot query at all and thus they cannot rediscover or retry.
>>>>>>
>>>>>> Does this make sense?
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>>>> Matthias,
>>>>>>>
>>>>>>> I try to clarify some concept.
>>>>>>>
>>>>>>> When streams state is REBALANCING, it means the user can just plain
>>>>>> retry.
>>>>>>>
>>>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
>>>> means
>>>>>>> state store migrated to another instance, the user needs to
>> rediscover
>>>>>> the
>>>>>>> store.
>>>>>>>
>>>>>>> Is my understanding correct?
>>>>>>>
>>>>>>>
>>>>>>> ---
>>>>>>> Vito
>>>>>>>
>>>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>>>> matthias@confluent.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the KIP Vito!
>>>>>>>>
>>>>>>>> I agree with what Guozhang said. The original idea of the Jira was,
>> to
>>>>>>>> give different exceptions for different "recovery" strategies to the
>>>>>> user.
>>>>>>>>
>>>>>>>> For example, if a store is currently recreated, a user just need to
>>>> wait
>>>>>>>> and can query the store later. On the other hand, if a store go
>>>> migrated
>>>>>>>> to another instance, a user needs to rediscover the store instead
>> of a
>>>>>>>> "plain retry".
>>>>>>>>
>>>>>>>> Fatal errors might be a third category.
>>>>>>>>
>>>>>>>> Not sure if there is something else?
>>>>>>>>
>>>>>>>> Anyway, the KIP should contain a section that talks about this ideas
>>>> and
>>>>>>>> reasoning.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>>>> Thanks for writing up the KIP.
>>>>>>>>>
>>>>>>>>> Vito, Matthias: one thing that I wanted to figure out first is what
>>>>>>>>> categories of errors we want to notify the users, if we only wants
>> to
>>>>>>>>> distinguish fatal v.s. retriable then probably we should rename the
>>>>>>>>> proposed StateStoreMigratedException / StateStoreClosedException
>>>>>> classes.
>>>>>>>>> And then from there we should list what are the possible internal
>>>>>>>>> exceptions ever thrown in those APIs in the call trace, and which
>>>>>>>>> exceptions should be wrapped to what others, and which ones should
>> be
>>>>>>>>> handled without re-throwing, and which ones should not be wrapped
>> at
>>>>>> all
>>>>>>>>> but directly thrown to user's face.
>>>>>>>>>
>>>>>>>>> Guozhang
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>>>
>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>>>>>>>>>>
>>>>>>>>>> Please have a look.
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
Thanks for the review.
I reply separately in the following sections.
---
Vito
On Sun, Apr 8, 2018 at 1:30 PM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for updating the KIP and sorry for the long pause...
>
> Seems you did a very thorough investigation of the code. It's useful to
> understand what user facing interfaces are affected.
(Some parts might be even too detailed for a KIP.)
>
I also think too detailed. Especially the section `Changes in call trace`.
Do you think it should be removed?
>
> To summarize my current understanding of your KIP, the main change is to
> introduce new exceptions that extend `InvalidStateStoreException`.
>
yep. Keep compatibility in this KIP is important things.
I think the best way is that all new exceptions extend from
`InvalidStateStoreException`.
>
> Some questions:
>
> - Why do we need ```? Could `InvalidStateStoreException` be used for
> this purpose?
>
Does this question miss some word?
>
> - What the superclass of `StateStoreStreamThreadNotRunningException`
> is? Should it be `InvalidStateStoreException` or `StateStoreFailException`
> ?
>
> - Is `StateStoreClosed` a fatal or retryable exception ?
>
>
I apologize for not well written parts. I tried to modify some code in the
recent period and modify the KIP.
The modification is now complete. Please look again.
>
>
> -Matthias
>
>
> On 2/21/18 5:15 PM, vito jeng wrote:
> > Matthias,
> >
> > Sorry for not response these days.
> > I just finished it. Please have a look. :)
> >
> >
> >
> > ---
> > Vito
> >
> > On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Is there any update on this KIP?
> >>
> >> -Matthias
> >>
> >> On 1/3/18 12:59 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> Thank you for your response.
> >>>
> >>> I think you are right. We need to look at the state both of
> >>> KafkaStreams and StreamThread.
> >>>
> >>> After further understanding of KafkaStreams thread and state store,
> >>> I am currently rewriting the KIP.
> >>>
> >>>
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <
> matthias@confluent.io>
> >>> wrote:
> >>>
> >>>> Vito,
> >>>>
> >>>> Sorry for this late reply.
> >>>>
> >>>> There can be two cases:
> >>>>
> >>>> - either a store got migrated way and thus, is not hosted an the
> >>>> application instance anymore,
> >>>> - or, a store is hosted but the instance is in state rebalance
> >>>>
> >>>> For the first case, users need to rediscover the store. For the second
> >>>> case, they need to wait until rebalance is finished.
> >>>>
> >>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
> >>>> uses cannot query at all and thus they cannot rediscover or retry.
> >>>>
> >>>> Does this make sense?
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 12/20/17 12:54 AM, vito jeng wrote:
> >>>>> Matthias,
> >>>>>
> >>>>> I try to clarify some concept.
> >>>>>
> >>>>> When streams state is REBALANCING, it means the user can just plain
> >>>> retry.
> >>>>>
> >>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
> >> means
> >>>>> state store migrated to another instance, the user needs to
> rediscover
> >>>> the
> >>>>> store.
> >>>>>
> >>>>> Is my understanding correct?
> >>>>>
> >>>>>
> >>>>> ---
> >>>>> Vito
> >>>>>
> >>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> >> matthias@confluent.io>
> >>>>> wrote:
> >>>>>
> >>>>>> Thanks for the KIP Vito!
> >>>>>>
> >>>>>> I agree with what Guozhang said. The original idea of the Jira was,
> to
> >>>>>> give different exceptions for different "recovery" strategies to the
> >>>> user.
> >>>>>>
> >>>>>> For example, if a store is currently recreated, a user just need to
> >> wait
> >>>>>> and can query the store later. On the other hand, if a store go
> >> migrated
> >>>>>> to another instance, a user needs to rediscover the store instead
> of a
> >>>>>> "plain retry".
> >>>>>>
> >>>>>> Fatal errors might be a third category.
> >>>>>>
> >>>>>> Not sure if there is something else?
> >>>>>>
> >>>>>> Anyway, the KIP should contain a section that talks about this ideas
> >> and
> >>>>>> reasoning.
> >>>>>>
> >>>>>>
> >>>>>> -Matthias
> >>>>>>
> >>>>>>
> >>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>>>> Thanks for writing up the KIP.
> >>>>>>>
> >>>>>>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>>>>>> categories of errors we want to notify the users, if we only wants
> to
> >>>>>>> distinguish fatal v.s. retriable then probably we should rename the
> >>>>>>> proposed StateStoreMigratedException / StateStoreClosedException
> >>>> classes.
> >>>>>>> And then from there we should list what are the possible internal
> >>>>>>> exceptions ever thrown in those APIs in the call trace, and which
> >>>>>>> exceptions should be wrapped to what others, and which ones should
> be
> >>>>>>> handled without re-throwing, and which ones should not be wrapped
> at
> >>>> all
> >>>>>>> but directly thrown to user's face.
> >>>>>>>
> >>>>>>> Guozhang
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I'd like to start discuss KIP-216:
> >>>>>>>>
> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>>>>>
> >>>>>>>> Please have a look.
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>> ---
> >>>>>>>> Vito
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks for updating the KIP and sorry for the long pause...
Seems you did a very thorough investigation of the code. It's useful to
understand what user facing interfaces are affected.
(Some parts might be even too detailed for a KIP.)
To summarize my current understanding of your KIP, the main change is to
introduce new exceptions that extend `InvalidStateStoreException`.
Some questions:
- Why do we need ```? Could `InvalidStateStoreException` be used for
this purpose?
- What the superclass of `StateStoreStreamThreadNotRunningException`
is? Should it be `InvalidStateStoreException` or `StateStoreFailException` ?
- Is `StateStoreClosed` a fatal or retryable exception ?
-Matthias
On 2/21/18 5:15 PM, vito jeng wrote:
> Matthias,
>
> Sorry for not response these days.
> I just finished it. Please have a look. :)
>
>
>
> ---
> Vito
>
> On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Is there any update on this KIP?
>>
>> -Matthias
>>
>> On 1/3/18 12:59 AM, vito jeng wrote:
>>> Matthias,
>>>
>>> Thank you for your response.
>>>
>>> I think you are right. We need to look at the state both of
>>> KafkaStreams and StreamThread.
>>>
>>> After further understanding of KafkaStreams thread and state store,
>>> I am currently rewriting the KIP.
>>>
>>>
>>>
>>>
>>> ---
>>> Vito
>>>
>>> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Vito,
>>>>
>>>> Sorry for this late reply.
>>>>
>>>> There can be two cases:
>>>>
>>>> - either a store got migrated way and thus, is not hosted an the
>>>> application instance anymore,
>>>> - or, a store is hosted but the instance is in state rebalance
>>>>
>>>> For the first case, users need to rediscover the store. For the second
>>>> case, they need to wait until rebalance is finished.
>>>>
>>>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
>>>> uses cannot query at all and thus they cannot rediscover or retry.
>>>>
>>>> Does this make sense?
>>>>
>>>> -Matthias
>>>>
>>>> On 12/20/17 12:54 AM, vito jeng wrote:
>>>>> Matthias,
>>>>>
>>>>> I try to clarify some concept.
>>>>>
>>>>> When streams state is REBALANCING, it means the user can just plain
>>>> retry.
>>>>>
>>>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
>> means
>>>>> state store migrated to another instance, the user needs to rediscover
>>>> the
>>>>> store.
>>>>>
>>>>> Is my understanding correct?
>>>>>
>>>>>
>>>>> ---
>>>>> Vito
>>>>>
>>>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
>> matthias@confluent.io>
>>>>> wrote:
>>>>>
>>>>>> Thanks for the KIP Vito!
>>>>>>
>>>>>> I agree with what Guozhang said. The original idea of the Jira was, to
>>>>>> give different exceptions for different "recovery" strategies to the
>>>> user.
>>>>>>
>>>>>> For example, if a store is currently recreated, a user just need to
>> wait
>>>>>> and can query the store later. On the other hand, if a store go
>> migrated
>>>>>> to another instance, a user needs to rediscover the store instead of a
>>>>>> "plain retry".
>>>>>>
>>>>>> Fatal errors might be a third category.
>>>>>>
>>>>>> Not sure if there is something else?
>>>>>>
>>>>>> Anyway, the KIP should contain a section that talks about this ideas
>> and
>>>>>> reasoning.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>>>> Thanks for writing up the KIP.
>>>>>>>
>>>>>>> Vito, Matthias: one thing that I wanted to figure out first is what
>>>>>>> categories of errors we want to notify the users, if we only wants to
>>>>>>> distinguish fatal v.s. retriable then probably we should rename the
>>>>>>> proposed StateStoreMigratedException / StateStoreClosedException
>>>> classes.
>>>>>>> And then from there we should list what are the possible internal
>>>>>>> exceptions ever thrown in those APIs in the call trace, and which
>>>>>>> exceptions should be wrapped to what others, and which ones should be
>>>>>>> handled without re-throwing, and which ones should not be wrapped at
>>>> all
>>>>>>> but directly thrown to user's face.
>>>>>>>
>>>>>>> Guozhang
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'd like to start discuss KIP-216:
>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>>>>>>>>
>>>>>>>> Please have a look.
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> ---
>>>>>>>> Vito
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
Sorry for not response these days.
I just finished it. Please have a look. :)
---
Vito
On Wed, Feb 14, 2018 at 5:45 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Is there any update on this KIP?
>
> -Matthias
>
> On 1/3/18 12:59 AM, vito jeng wrote:
> > Matthias,
> >
> > Thank you for your response.
> >
> > I think you are right. We need to look at the state both of
> > KafkaStreams and StreamThread.
> >
> > After further understanding of KafkaStreams thread and state store,
> > I am currently rewriting the KIP.
> >
> >
> >
> >
> > ---
> > Vito
> >
> > On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Vito,
> >>
> >> Sorry for this late reply.
> >>
> >> There can be two cases:
> >>
> >> - either a store got migrated way and thus, is not hosted an the
> >> application instance anymore,
> >> - or, a store is hosted but the instance is in state rebalance
> >>
> >> For the first case, users need to rediscover the store. For the second
> >> case, they need to wait until rebalance is finished.
> >>
> >> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
> >> uses cannot query at all and thus they cannot rediscover or retry.
> >>
> >> Does this make sense?
> >>
> >> -Matthias
> >>
> >> On 12/20/17 12:54 AM, vito jeng wrote:
> >>> Matthias,
> >>>
> >>> I try to clarify some concept.
> >>>
> >>> When streams state is REBALANCING, it means the user can just plain
> >> retry.
> >>>
> >>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it
> means
> >>> state store migrated to another instance, the user needs to rediscover
> >> the
> >>> store.
> >>>
> >>> Is my understanding correct?
> >>>
> >>>
> >>> ---
> >>> Vito
> >>>
> >>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <
> matthias@confluent.io>
> >>> wrote:
> >>>
> >>>> Thanks for the KIP Vito!
> >>>>
> >>>> I agree with what Guozhang said. The original idea of the Jira was, to
> >>>> give different exceptions for different "recovery" strategies to the
> >> user.
> >>>>
> >>>> For example, if a store is currently recreated, a user just need to
> wait
> >>>> and can query the store later. On the other hand, if a store go
> migrated
> >>>> to another instance, a user needs to rediscover the store instead of a
> >>>> "plain retry".
> >>>>
> >>>> Fatal errors might be a third category.
> >>>>
> >>>> Not sure if there is something else?
> >>>>
> >>>> Anyway, the KIP should contain a section that talks about this ideas
> and
> >>>> reasoning.
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>>
> >>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>>>> Thanks for writing up the KIP.
> >>>>>
> >>>>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>>>> categories of errors we want to notify the users, if we only wants to
> >>>>> distinguish fatal v.s. retriable then probably we should rename the
> >>>>> proposed StateStoreMigratedException / StateStoreClosedException
> >> classes.
> >>>>> And then from there we should list what are the possible internal
> >>>>> exceptions ever thrown in those APIs in the call trace, and which
> >>>>> exceptions should be wrapped to what others, and which ones should be
> >>>>> handled without re-throwing, and which ones should not be wrapped at
> >> all
> >>>>> but directly thrown to user's face.
> >>>>>
> >>>>> Guozhang
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
> >> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I'd like to start discuss KIP-216:
> >>>>>>
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>>>
> >>>>>> Please have a look.
> >>>>>> Thanks!
> >>>>>>
> >>>>>> ---
> >>>>>> Vito
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Is there any update on this KIP?
-Matthias
On 1/3/18 12:59 AM, vito jeng wrote:
> Matthias,
>
> Thank you for your response.
>
> I think you are right. We need to look at the state both of
> KafkaStreams and StreamThread.
>
> After further understanding of KafkaStreams thread and state store,
> I am currently rewriting the KIP.
>
>
>
>
> ---
> Vito
>
> On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Vito,
>>
>> Sorry for this late reply.
>>
>> There can be two cases:
>>
>> - either a store got migrated way and thus, is not hosted an the
>> application instance anymore,
>> - or, a store is hosted but the instance is in state rebalance
>>
>> For the first case, users need to rediscover the store. For the second
>> case, they need to wait until rebalance is finished.
>>
>> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
>> uses cannot query at all and thus they cannot rediscover or retry.
>>
>> Does this make sense?
>>
>> -Matthias
>>
>> On 12/20/17 12:54 AM, vito jeng wrote:
>>> Matthias,
>>>
>>> I try to clarify some concept.
>>>
>>> When streams state is REBALANCING, it means the user can just plain
>> retry.
>>>
>>> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it means
>>> state store migrated to another instance, the user needs to rediscover
>> the
>>> store.
>>>
>>> Is my understanding correct?
>>>
>>>
>>> ---
>>> Vito
>>>
>>> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Thanks for the KIP Vito!
>>>>
>>>> I agree with what Guozhang said. The original idea of the Jira was, to
>>>> give different exceptions for different "recovery" strategies to the
>> user.
>>>>
>>>> For example, if a store is currently recreated, a user just need to wait
>>>> and can query the store later. On the other hand, if a store go migrated
>>>> to another instance, a user needs to rediscover the store instead of a
>>>> "plain retry".
>>>>
>>>> Fatal errors might be a third category.
>>>>
>>>> Not sure if there is something else?
>>>>
>>>> Anyway, the KIP should contain a section that talks about this ideas and
>>>> reasoning.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>>>> Thanks for writing up the KIP.
>>>>>
>>>>> Vito, Matthias: one thing that I wanted to figure out first is what
>>>>> categories of errors we want to notify the users, if we only wants to
>>>>> distinguish fatal v.s. retriable then probably we should rename the
>>>>> proposed StateStoreMigratedException / StateStoreClosedException
>> classes.
>>>>> And then from there we should list what are the possible internal
>>>>> exceptions ever thrown in those APIs in the call trace, and which
>>>>> exceptions should be wrapped to what others, and which ones should be
>>>>> handled without re-throwing, and which ones should not be wrapped at
>> all
>>>>> but directly thrown to user's face.
>>>>>
>>>>> Guozhang
>>>>>
>>>>>
>>>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'd like to start discuss KIP-216:
>>>>>>
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>>>>>>
>>>>>> Please have a look.
>>>>>> Thanks!
>>>>>>
>>>>>> ---
>>>>>> Vito
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
Thank you for your response.
I think you are right. We need to look at the state both of
KafkaStreams and StreamThread.
After further understanding of KafkaStreams thread and state store,
I am currently rewriting the KIP.
---
Vito
On Fri, Dec 29, 2017 at 4:32 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Vito,
>
> Sorry for this late reply.
>
> There can be two cases:
>
> - either a store got migrated way and thus, is not hosted an the
> application instance anymore,
> - or, a store is hosted but the instance is in state rebalance
>
> For the first case, users need to rediscover the store. For the second
> case, they need to wait until rebalance is finished.
>
> If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
> uses cannot query at all and thus they cannot rediscover or retry.
>
> Does this make sense?
>
> -Matthias
>
> On 12/20/17 12:54 AM, vito jeng wrote:
> > Matthias,
> >
> > I try to clarify some concept.
> >
> > When streams state is REBALANCING, it means the user can just plain
> retry.
> >
> > When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it means
> > state store migrated to another instance, the user needs to rediscover
> the
> > store.
> >
> > Is my understanding correct?
> >
> >
> > ---
> > Vito
> >
> > On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> Thanks for the KIP Vito!
> >>
> >> I agree with what Guozhang said. The original idea of the Jira was, to
> >> give different exceptions for different "recovery" strategies to the
> user.
> >>
> >> For example, if a store is currently recreated, a user just need to wait
> >> and can query the store later. On the other hand, if a store go migrated
> >> to another instance, a user needs to rediscover the store instead of a
> >> "plain retry".
> >>
> >> Fatal errors might be a third category.
> >>
> >> Not sure if there is something else?
> >>
> >> Anyway, the KIP should contain a section that talks about this ideas and
> >> reasoning.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> >>> Thanks for writing up the KIP.
> >>>
> >>> Vito, Matthias: one thing that I wanted to figure out first is what
> >>> categories of errors we want to notify the users, if we only wants to
> >>> distinguish fatal v.s. retriable then probably we should rename the
> >>> proposed StateStoreMigratedException / StateStoreClosedException
> classes.
> >>> And then from there we should list what are the possible internal
> >>> exceptions ever thrown in those APIs in the call trace, and which
> >>> exceptions should be wrapped to what others, and which ones should be
> >>> handled without re-throwing, and which ones should not be wrapped at
> all
> >>> but directly thrown to user's face.
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw>
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'd like to start discuss KIP-216:
> >>>>
> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>>>
> >>>> Please have a look.
> >>>> Thanks!
> >>>>
> >>>> ---
> >>>> Vito
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Vito,
Sorry for this late reply.
There can be two cases:
- either a store got migrated way and thus, is not hosted an the
application instance anymore,
- or, a store is hosted but the instance is in state rebalance
For the first case, users need to rediscover the store. For the second
case, they need to wait until rebalance is finished.
If KafkaStreams is in state ERROR, PENDING_SHUTDOWN, or NOT_RUNNING,
uses cannot query at all and thus they cannot rediscover or retry.
Does this make sense?
-Matthias
On 12/20/17 12:54 AM, vito jeng wrote:
> Matthias,
>
> I try to clarify some concept.
>
> When streams state is REBALANCING, it means the user can just plain retry.
>
> When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it means
> state store migrated to another instance, the user needs to rediscover the
> store.
>
> Is my understanding correct?
>
>
> ---
> Vito
>
> On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Thanks for the KIP Vito!
>>
>> I agree with what Guozhang said. The original idea of the Jira was, to
>> give different exceptions for different "recovery" strategies to the user.
>>
>> For example, if a store is currently recreated, a user just need to wait
>> and can query the store later. On the other hand, if a store go migrated
>> to another instance, a user needs to rediscover the store instead of a
>> "plain retry".
>>
>> Fatal errors might be a third category.
>>
>> Not sure if there is something else?
>>
>> Anyway, the KIP should contain a section that talks about this ideas and
>> reasoning.
>>
>>
>> -Matthias
>>
>>
>> On 11/3/17 11:26 PM, Guozhang Wang wrote:
>>> Thanks for writing up the KIP.
>>>
>>> Vito, Matthias: one thing that I wanted to figure out first is what
>>> categories of errors we want to notify the users, if we only wants to
>>> distinguish fatal v.s. retriable then probably we should rename the
>>> proposed StateStoreMigratedException / StateStoreClosedException classes.
>>> And then from there we should list what are the possible internal
>>> exceptions ever thrown in those APIs in the call trace, and which
>>> exceptions should be wrapped to what others, and which ones should be
>>> handled without re-throwing, and which ones should not be wrapped at all
>>> but directly thrown to user's face.
>>>
>>> Guozhang
>>>
>>>
>>> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'd like to start discuss KIP-216:
>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>>>>
>>>> Please have a look.
>>>> Thanks!
>>>>
>>>> ---
>>>> Vito
>>>>
>>>
>>>
>>>
>>
>>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by vito jeng <vi...@is-land.com.tw>.
Matthias,
I try to clarify some concept.
When streams state is REBALANCING, it means the user can just plain retry.
When streams state is ERROR or PENDING_SHUTDOWN or NOT_RUNNING, it means
state store migrated to another instance, the user needs to rediscover the
store.
Is my understanding correct?
---
Vito
On Sun, Nov 5, 2017 at 12:30 AM, Matthias J. Sax <ma...@confluent.io>
wrote:
> Thanks for the KIP Vito!
>
> I agree with what Guozhang said. The original idea of the Jira was, to
> give different exceptions for different "recovery" strategies to the user.
>
> For example, if a store is currently recreated, a user just need to wait
> and can query the store later. On the other hand, if a store go migrated
> to another instance, a user needs to rediscover the store instead of a
> "plain retry".
>
> Fatal errors might be a third category.
>
> Not sure if there is something else?
>
> Anyway, the KIP should contain a section that talks about this ideas and
> reasoning.
>
>
> -Matthias
>
>
> On 11/3/17 11:26 PM, Guozhang Wang wrote:
> > Thanks for writing up the KIP.
> >
> > Vito, Matthias: one thing that I wanted to figure out first is what
> > categories of errors we want to notify the users, if we only wants to
> > distinguish fatal v.s. retriable then probably we should rename the
> > proposed StateStoreMigratedException / StateStoreClosedException classes.
> > And then from there we should list what are the possible internal
> > exceptions ever thrown in those APIs in the call trace, and which
> > exceptions should be wrapped to what others, and which ones should be
> > handled without re-throwing, and which ones should not be wrapped at all
> > but directly thrown to user's face.
> >
> > Guozhang
> >
> >
> > On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw> wrote:
> >
> >> Hi,
> >>
> >> I'd like to start discuss KIP-216:
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
> >>
> >> Please have a look.
> >> Thanks!
> >>
> >> ---
> >> Vito
> >>
> >
> >
> >
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Thanks for the KIP Vito!
I agree with what Guozhang said. The original idea of the Jira was, to
give different exceptions for different "recovery" strategies to the user.
For example, if a store is currently recreated, a user just need to wait
and can query the store later. On the other hand, if a store go migrated
to another instance, a user needs to rediscover the store instead of a
"plain retry".
Fatal errors might be a third category.
Not sure if there is something else?
Anyway, the KIP should contain a section that talks about this ideas and
reasoning.
-Matthias
On 11/3/17 11:26 PM, Guozhang Wang wrote:
> Thanks for writing up the KIP.
>
> Vito, Matthias: one thing that I wanted to figure out first is what
> categories of errors we want to notify the users, if we only wants to
> distinguish fatal v.s. retriable then probably we should rename the
> proposed StateStoreMigratedException / StateStoreClosedException classes.
> And then from there we should list what are the possible internal
> exceptions ever thrown in those APIs in the call trace, and which
> exceptions should be wrapped to what others, and which ones should be
> handled without re-throwing, and which ones should not be wrapped at all
> but directly thrown to user's face.
>
> Guozhang
>
>
> On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw> wrote:
>
>> Hi,
>>
>> I'd like to start discuss KIP-216:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>>
>> Please have a look.
>> Thanks!
>>
>> ---
>> Vito
>>
>
>
>
Re: [DISCUSS]KIP-216: IQ should throw different exceptions for
different errors
Posted by Guozhang Wang <wa...@gmail.com>.
Thanks for writing up the KIP.
Vito, Matthias: one thing that I wanted to figure out first is what
categories of errors we want to notify the users, if we only wants to
distinguish fatal v.s. retriable then probably we should rename the
proposed StateStoreMigratedException / StateStoreClosedException classes.
And then from there we should list what are the possible internal
exceptions ever thrown in those APIs in the call trace, and which
exceptions should be wrapped to what others, and which ones should be
handled without re-throwing, and which ones should not be wrapped at all
but directly thrown to user's face.
Guozhang
On Wed, Nov 1, 2017 at 11:09 PM, vito jeng <vi...@is-land.com.tw> wrote:
> Hi,
>
> I'd like to start discuss KIP-216:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 216%3A+IQ+should+throw+different+exceptions+for+different+errors
>
> Please have a look.
> Thanks!
>
> ---
> Vito
>
--
-- Guozhang