You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Ryan Ruel <r....@icloud.com> on 2023/05/19 11:44:07 UTC
ModeledFramework/Async Error Handling Expectations
I’ve encountered an issue in my application where I don’t seem to be receiving exceptions back from Curator in certain error scenarios.
My application:
* Has a pool of worker threads working on different jobs which need to read/write ZNodes in ZK.
* Utilizes the Curator ModeledFramework to serialize data within the ZNodes.
* As this is using ModeledFramework (which is built upon Curator Async), the application uses a Future.get() to wait for Curator to respond with either a success or failure result for each operation.
Under heavy load, where the ZK connectivity becomes flakey, I occasionally encounter a case where all my worker threads block on calls to Future.get().
With a connection loss event occurs (or if ZK is just too busy to reply in a timely manner), I’d expect to see exceptions thrown by Curator, but this never happens… the application threads wedge indefinitely.
Is the expectation when using the Async APIs that we should always expect a success/failure response?
Or is the expectation that the application should implement an additional timer in the event that Curator doesn’t respond?
If it’s the former, I can dig further into why Curator is not responding.
/Ryan
----
Ryan Ruel
ryan@ryanruel.com
Re: ModeledFramework/Async Error Handling Expectations
Posted by Kezhu Wang <ke...@gmail.com>.
Hi Ryan,
I have opened https://github.com/apache/curator/pull/464 in case you
are interested.
Best,
Kezhu Wang
On Mon, May 22, 2023 at 6:28 PM Ryan Ruel <r....@icloud.com> wrote:
>
> We’ve been using 5.5.0.
>
> That does look suspiciously close to what we are seeing!
>
> /Ryan
>
> > On May 22, 2023, at 2:54 AM, Kezhu Wang <ke...@gmail.com> wrote:
> >
> > Hi Ryan,
> >
> > Which curator version did you use ? Is it possibly covered by
> > CURATOR-673[1] (Complete BackgroundCallback if curator got closed or
> > exceptions from no-zookeeper world) ?
> >
> > [1]: https://issues.apache.org/jira/browse/CURATOR-673
> >
> >
> > Best,
> > Kezhu Wang
> >
> > On Sat, May 20, 2023 at 12:44 PM tison <wa...@gmail.com> wrote:
> >>
> >> Yep.
> >>
> >> You're welcome to file a ticket on our JIRA project to share your reproductive code - https://issues.apache.org/jira/projects/CURATOR
> >>
> >> If you don't have a JIRA account yet, you can self-request at https://selfserve.apache.org/jira-account.html
> >>
> >> Best,
> >> tison.
> >>
> >>
> >> Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 22:38写道：
> >>>
> >>> It will require some effort to get there (without exposing proprietary code), but the goal would be reproduction (hopefully via unit test).
> >>>
> >>> Am I correct then in that these conditions are unexpected?
> >>>
> >>> ----
> >>> Ryan Ruel
> >>> ryan@ryanruel.com
> >>>
> >>> On May 19, 2023, at 9:33 AM, tison <wa...@gmail.com> wrote:
> >>>
> >>>
> >>> Hi Ryan,
> >>>
> >>> Thanks for reporting your use case!
> >>>
> >>> According to your description, it's hard to investigate which part can be the root cause of the hanging result. Could you provide a reproduce code repo, or share the significant code snippet with related logs?
> >>>
> >>> Best,
> >>> tison.
> >>>
> >>>
> >>> Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 19:44写道：
> >>>>
> >>>> I’ve encountered an issue in my application where I don’t seem to be receiving exceptions back from Curator in certain error scenarios.
> >>>>
> >>>> My application:
> >>>> * Has a pool of worker threads working on different jobs which need to read/write ZNodes in ZK.
> >>>> * Utilizes the Curator ModeledFramework to serialize data within the ZNodes.
> >>>> * As this is using ModeledFramework (which is built upon Curator Async), the application uses a Future.get() to wait for Curator to respond with either a success or failure result for each operation.
> >>>>
> >>>> Under heavy load, where the ZK connectivity becomes flakey, I occasionally encounter a case where all my worker threads block on calls to Future.get().
> >>>>
> >>>> With a connection loss event occurs (or if ZK is just too busy to reply in a timely manner), I’d expect to see exceptions thrown by Curator, but this never happens… the application threads wedge indefinitely.
> >>>>
> >>>> Is the expectation when using the Async APIs that we should always expect a success/failure response?
> >>>>
> >>>> Or is the expectation that the application should implement an additional timer in the event that Curator doesn’t respond?
> >>>>
> >>>> If it’s the former, I can dig further into why Curator is not responding.
> >>>>
> >>>> /Ryan
> >>>>
> >>>> ----
> >>>> Ryan Ruel
> >>>> ryan@ryanruel.com
>
Re: ModeledFramework/Async Error Handling Expectations
Posted by Ryan Ruel <r....@icloud.com>.
We’ve been using 5.5.0.
That does look suspiciously close to what we are seeing!
/Ryan
> On May 22, 2023, at 2:54 AM, Kezhu Wang <ke...@gmail.com> wrote:
>
> Hi Ryan,
>
> Which curator version did you use ? Is it possibly covered by
> CURATOR-673[1] (Complete BackgroundCallback if curator got closed or
> exceptions from no-zookeeper world) ?
>
> [1]: https://issues.apache.org/jira/browse/CURATOR-673
>
>
> Best,
> Kezhu Wang
>
> On Sat, May 20, 2023 at 12:44 PM tison <wa...@gmail.com> wrote:
>>
>> Yep.
>>
>> You're welcome to file a ticket on our JIRA project to share your reproductive code - https://issues.apache.org/jira/projects/CURATOR
>>
>> If you don't have a JIRA account yet, you can self-request at https://selfserve.apache.org/jira-account.html
>>
>> Best,
>> tison.
>>
>>
>> Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 22:38写道：
>>>
>>> It will require some effort to get there (without exposing proprietary code), but the goal would be reproduction (hopefully via unit test).
>>>
>>> Am I correct then in that these conditions are unexpected?
>>>
>>> ----
>>> Ryan Ruel
>>> ryan@ryanruel.com
>>>
>>> On May 19, 2023, at 9:33 AM, tison <wa...@gmail.com> wrote:
>>>
>>>
>>> Hi Ryan,
>>>
>>> Thanks for reporting your use case!
>>>
>>> According to your description, it's hard to investigate which part can be the root cause of the hanging result. Could you provide a reproduce code repo, or share the significant code snippet with related logs?
>>>
>>> Best,
>>> tison.
>>>
>>>
>>> Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 19:44写道：
>>>>
>>>> I’ve encountered an issue in my application where I don’t seem to be receiving exceptions back from Curator in certain error scenarios.
>>>>
>>>> My application:
>>>> * Has a pool of worker threads working on different jobs which need to read/write ZNodes in ZK.
>>>> * Utilizes the Curator ModeledFramework to serialize data within the ZNodes.
>>>> * As this is using ModeledFramework (which is built upon Curator Async), the application uses a Future.get() to wait for Curator to respond with either a success or failure result for each operation.
>>>>
>>>> Under heavy load, where the ZK connectivity becomes flakey, I occasionally encounter a case where all my worker threads block on calls to Future.get().
>>>>
>>>> With a connection loss event occurs (or if ZK is just too busy to reply in a timely manner), I’d expect to see exceptions thrown by Curator, but this never happens… the application threads wedge indefinitely.
>>>>
>>>> Is the expectation when using the Async APIs that we should always expect a success/failure response?
>>>>
>>>> Or is the expectation that the application should implement an additional timer in the event that Curator doesn’t respond?
>>>>
>>>> If it’s the former, I can dig further into why Curator is not responding.
>>>>
>>>> /Ryan
>>>>
>>>> ----
>>>> Ryan Ruel
>>>> ryan@ryanruel.com
Re: ModeledFramework/Async Error Handling Expectations
Posted by Kezhu Wang <ke...@gmail.com>.
Hi Ryan,
Which curator version did you use ? Is it possibly covered by
CURATOR-673[1] (Complete BackgroundCallback if curator got closed or
exceptions from no-zookeeper world) ?
[1]: https://issues.apache.org/jira/browse/CURATOR-673
Best,
Kezhu Wang
On Sat, May 20, 2023 at 12:44 PM tison <wa...@gmail.com> wrote:
>
> Yep.
>
> You're welcome to file a ticket on our JIRA project to share your reproductive code - https://issues.apache.org/jira/projects/CURATOR
>
> If you don't have a JIRA account yet, you can self-request at https://selfserve.apache.org/jira-account.html
>
> Best,
> tison.
>
>
> Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 22:38写道：
>>
>> It will require some effort to get there (without exposing proprietary code), but the goal would be reproduction (hopefully via unit test).
>>
>> Am I correct then in that these conditions are unexpected?
>>
>> ----
>> Ryan Ruel
>> ryan@ryanruel.com
>>
>> On May 19, 2023, at 9:33 AM, tison <wa...@gmail.com> wrote:
>>
>>
>> Hi Ryan,
>>
>> Thanks for reporting your use case!
>>
>> According to your description, it's hard to investigate which part can be the root cause of the hanging result. Could you provide a reproduce code repo, or share the significant code snippet with related logs?
>>
>> Best,
>> tison.
>>
>>
>> Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 19:44写道：
>>>
>>> I’ve encountered an issue in my application where I don’t seem to be receiving exceptions back from Curator in certain error scenarios.
>>>
>>> My application:
>>> * Has a pool of worker threads working on different jobs which need to read/write ZNodes in ZK.
>>> * Utilizes the Curator ModeledFramework to serialize data within the ZNodes.
>>> * As this is using ModeledFramework (which is built upon Curator Async), the application uses a Future.get() to wait for Curator to respond with either a success or failure result for each operation.
>>>
>>> Under heavy load, where the ZK connectivity becomes flakey, I occasionally encounter a case where all my worker threads block on calls to Future.get().
>>>
>>> With a connection loss event occurs (or if ZK is just too busy to reply in a timely manner), I’d expect to see exceptions thrown by Curator, but this never happens… the application threads wedge indefinitely.
>>>
>>> Is the expectation when using the Async APIs that we should always expect a success/failure response?
>>>
>>> Or is the expectation that the application should implement an additional timer in the event that Curator doesn’t respond?
>>>
>>> If it’s the former, I can dig further into why Curator is not responding.
>>>
>>> /Ryan
>>>
>>> ----
>>> Ryan Ruel
>>> ryan@ryanruel.com
Re: ModeledFramework/Async Error Handling Expectations
Posted by tison <wa...@gmail.com>.
Yep.
You're welcome to file a ticket on our JIRA project to share your
reproductive code - https://issues.apache.org/jira/projects/CURATOR
If you don't have a JIRA account yet, you can self-request at
https://selfserve.apache.org/jira-account.html
Best,
tison.
Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 22:38写道：
> It will require some effort to get there (without exposing proprietary
> code), but the goal would be reproduction (hopefully via unit test).
>
> Am I correct then in that these conditions are unexpected?
>
> ----
> Ryan Ruel
> ryan@ryanruel.com
>
> On May 19, 2023, at 9:33 AM, tison <wa...@gmail.com> wrote:
>
>
> Hi Ryan,
>
> Thanks for reporting your use case!
>
> According to your description, it's hard to investigate which part can be
> the root cause of the hanging result. Could you provide a reproduce code
> repo, or share the significant code snippet with related logs?
>
> Best,
> tison.
>
>
> Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 19:44写道：
>
>> I’ve encountered an issue in my application where I don’t seem to be
>> receiving exceptions back from Curator in certain error scenarios.
>>
>> My application:
>> * Has a pool of worker threads working on different jobs which need to
>> read/write ZNodes in ZK.
>> * Utilizes the Curator ModeledFramework to serialize data within the
>> ZNodes.
>> * As this is using ModeledFramework (which is built upon Curator Async),
>> the application uses a Future.get() to wait for Curator to respond with
>> either a success or failure result for each operation.
>>
>> Under heavy load, where the ZK connectivity becomes flakey, I
>> occasionally encounter a case where all my worker threads block on calls to
>> Future.get().
>>
>> With a connection loss event occurs (or if ZK is just too busy to reply
>> in a timely manner), I’d expect to see exceptions thrown by Curator, but
>> this never happens… the application threads wedge indefinitely.
>>
>> Is the expectation when using the Async APIs that we should always expect
>> a success/failure response?
>>
>> Or is the expectation that the application should implement an additional
>> timer in the event that Curator doesn’t respond?
>>
>> If it’s the former, I can dig further into why Curator is not responding.
>>
>> /Ryan
>>
>> ----
>> Ryan Ruel
>> ryan@ryanruel.com
>
>
Re: ModeledFramework/Async Error Handling Expectations
Posted by Ryan Ruel <r....@icloud.com>.
It will require some effort to get there (without exposing proprietary code),
but the goal would be reproduction (hopefully via unit test).
Am I correct then in that these conditions are unexpected?
\----
Ryan Ruel
ryan@ryanruel.com
> On May 19, 2023, at 9:33 AM, tison <wa...@gmail.com> wrote:
>
>
>
>
> Hi Ryan,
>
>
>
>
> Thanks for reporting your use case!
>
>
>
>
> According to your description, it's hard to investigate which part can be
> the root cause of the hanging result. Could you provide a reproduce code
> repo, or share the significant code snippet with related logs?
>
>
>
>
> Best,
>
> tison.
>
>
>
>
>
>
>
> Ryan Ruel <[r.ruel@icloud.com](mailto:r.ruel@icloud.com)> 于2023年5月19日周五
> 19:44写道：
>
>
>> I’ve encountered an issue in my application where I don’t seem to be
receiving exceptions back from Curator in certain error scenarios.
>
> My application:
> * Has a pool of worker threads working on different jobs which need to
> read/write ZNodes in ZK.
> * Utilizes the Curator ModeledFramework to serialize data within the
> ZNodes.
> * As this is using ModeledFramework (which is built upon Curator Async),
> the application uses a Future.get() to wait for Curator to respond with
> either a success or failure result for each operation.
>
> Under heavy load, where the ZK connectivity becomes flakey, I occasionally
> encounter a case where all my worker threads block on calls to Future.get().
>
> With a connection loss event occurs (or if ZK is just too busy to reply in
> a timely manner), I’d expect to see exceptions thrown by Curator, but this
> never happens… the application threads wedge indefinitely.
>
> Is the expectation when using the Async APIs that we should always expect a
> success/failure response?
>
> Or is the expectation that the application should implement an additional
> timer in the event that Curator doesn’t respond?
>
> If it’s the former, I can dig further into why Curator is not responding.
>
> /Ryan
>
> \----
> Ryan Ruel
> [ryan@ryanruel.com](mailto:ryan@ryanruel.com)
Re: ModeledFramework/Async Error Handling Expectations
Posted by tison <wa...@gmail.com>.
Hi Ryan,
Thanks for reporting your use case!
According to your description, it's hard to investigate which part can be
the root cause of the hanging result. Could you provide a reproduce code
repo, or share the significant code snippet with related logs?
Best,
tison.
Ryan Ruel <r....@icloud.com> 于2023年5月19日周五 19:44写道：
> I’ve encountered an issue in my application where I don’t seem to be
> receiving exceptions back from Curator in certain error scenarios.
>
> My application:
> * Has a pool of worker threads working on different jobs which need to
> read/write ZNodes in ZK.
> * Utilizes the Curator ModeledFramework to serialize data within the
> ZNodes.
> * As this is using ModeledFramework (which is built upon Curator Async),
> the application uses a Future.get() to wait for Curator to respond with
> either a success or failure result for each operation.
>
> Under heavy load, where the ZK connectivity becomes flakey, I occasionally
> encounter a case where all my worker threads block on calls to Future.get().
>
> With a connection loss event occurs (or if ZK is just too busy to reply in
> a timely manner), I’d expect to see exceptions thrown by Curator, but this
> never happens… the application threads wedge indefinitely.
>
> Is the expectation when using the Async APIs that we should always expect
> a success/failure response?
>
> Or is the expectation that the application should implement an additional
> timer in the event that Curator doesn’t respond?
>
> If it’s the former, I can dig further into why Curator is not responding.
>
> /Ryan
>
> ----
> Ryan Ruel
> ryan@ryanruel.com