You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Martin Serrano <ma...@attivio.com> on 2011/08/15 23:15:25 UTC
thread pool executor for event dispatch
Hi,
Has the zookeeper community considered adding an option to use a thread pool executor for event dispatch? One pattern we have developed for our library is to trigger event listeners in a separate thread to guard against blocking operations in listeners. A blocking watcher will prevent new events/watchers from being triggered. By adding the ability to configure which Executor was used by a zookeeper client for event delivery library writers could make the choice that is best for them. Being able to do this at the client level (rather than for each individual watcher class I implement) is a cleaner approach IMO.
Thoughts?
-Martin
RE: thread pool executor for event dispatch
Posted by Martin Serrano <ma...@attivio.com>.
I understand the guarantees and the impact a thread pool executor would have. We are using single thread executor with unbounded queue for just that reason (for one watcher type). I could see scenarios where a particular client did not need those guarantees. In that case the event message traffic would not be bounded by the time to handle the events. But these are subtle issues and I can understand not wanting to provide this much rope. A motivated user can just create a wrapper around their watchers that does whatever they want. Just thought I'd see what others thought.
-Martin
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Monday, August 15, 2011 6:42 PM
To: user@zookeeper.apache.org
Subject: Re: thread pool executor for event dispatch
And to add to this, the ordering guarantees that ZK provides are perhaps the most important of all. They are what allows proper reasoning.
The application can always push work into an execution pool from the actual watcher if threaded response makes sense. That avoids blocking the notification thread and allows whatever level of concurrency is appropriate.
On Mon, Aug 15, 2011 at 3:38 PM, Mahadev Konar <ma...@hortonworks.com>wrote:
> Martin,
> The problem with using a thread pool is that it becomes difficult to
> make sense about order of events that occur in the system.
>
> Here is the snippet from ZK wiki on client guarantees:
>
> What ZooKeeper Guarantees about Watches With regard to watches,
> ZooKeeper maintains these guarantees:
>
> • Watches are ordered with respect to other events, other
> watches, and asynchronous replies. The ZooKeeper client libraries
> ensures that everything is dispatched in order.
> • A client will see a watch event for a znode it is watching
> before seeing the new data that corresponds to that znode.
> • The order of watch events from ZooKeeper corresponds to the
> order of the updates as seen by the ZooKeeper service.
>
>
> hope that helps
> mahadev
>
> On Aug 15, 2011, at 2:15 PM, Martin Serrano wrote:
>
> > Hi,
> >
> > Has the zookeeper community considered adding an option to use a
> > thread
> pool executor for event dispatch? One pattern we have developed for
> our library is to trigger event listeners in a separate thread to
> guard against blocking operations in listeners. A blocking watcher
> will prevent new events/watchers from being triggered. By adding the
> ability to configure which Executor was used by a zookeeper client for
> event delivery library writers could make the choice that is best for
> them. Being able to do this at the client level (rather than for each
> individual watcher class I
> implement) is a cleaner approach IMO.
> >
> > Thoughts?
> >
> > -Martin
>
>
Re: thread pool executor for event dispatch
Posted by Ted Dunning <te...@gmail.com>.
And to add to this, the ordering guarantees that ZK provides are perhaps the
most important of all. They are what allows proper reasoning.
The application can always push work into an execution pool from the actual
watcher if threaded response makes sense. That avoids blocking the
notification thread and allows whatever level of concurrency is appropriate.
On Mon, Aug 15, 2011 at 3:38 PM, Mahadev Konar <ma...@hortonworks.com>wrote:
> Martin,
> The problem with using a thread pool is that it becomes difficult to make
> sense about order of events that occur in the system.
>
> Here is the snippet from ZK wiki on client guarantees:
>
> What ZooKeeper Guarantees about Watches
> With regard to watches, ZooKeeper maintains these guarantees:
>
> • Watches are ordered with respect to other events, other watches,
> and asynchronous replies. The ZooKeeper client libraries ensures that
> everything is dispatched in order.
> • A client will see a watch event for a znode it is watching before
> seeing the new data that corresponds to that znode.
> • The order of watch events from ZooKeeper corresponds to the order
> of the updates as seen by the ZooKeeper service.
>
>
> hope that helps
> mahadev
>
> On Aug 15, 2011, at 2:15 PM, Martin Serrano wrote:
>
> > Hi,
> >
> > Has the zookeeper community considered adding an option to use a thread
> pool executor for event dispatch? One pattern we have developed for our
> library is to trigger event listeners in a separate thread to guard against
> blocking operations in listeners. A blocking watcher will prevent new
> events/watchers from being triggered. By adding the ability to configure
> which Executor was used by a zookeeper client for event delivery library
> writers could make the choice that is best for them. Being able to do this
> at the client level (rather than for each individual watcher class I
> implement) is a cleaner approach IMO.
> >
> > Thoughts?
> >
> > -Martin
>
>
Re: thread pool executor for event dispatch
Posted by Mahadev Konar <ma...@hortonworks.com>.
Martin,
The problem with using a thread pool is that it becomes difficult to make sense about order of events that occur in the system.
Here is the snippet from ZK wiki on client guarantees:
What ZooKeeper Guarantees about Watches
With regard to watches, ZooKeeper maintains these guarantees:
• Watches are ordered with respect to other events, other watches, and asynchronous replies. The ZooKeeper client libraries ensures that everything is dispatched in order.
• A client will see a watch event for a znode it is watching before seeing the new data that corresponds to that znode.
• The order of watch events from ZooKeeper corresponds to the order of the updates as seen by the ZooKeeper service.
hope that helps
mahadev
On Aug 15, 2011, at 2:15 PM, Martin Serrano wrote:
> Hi,
>
> Has the zookeeper community considered adding an option to use a thread pool executor for event dispatch? One pattern we have developed for our library is to trigger event listeners in a separate thread to guard against blocking operations in listeners. A blocking watcher will prevent new events/watchers from being triggered. By adding the ability to configure which Executor was used by a zookeeper client for event delivery library writers could make the choice that is best for them. Being able to do this at the client level (rather than for each individual watcher class I implement) is a cleaner approach IMO.
>
> Thoughts?
>
> -Martin
Re: thread pool executor for event dispatch
Posted by Ted Dunning <te...@gmail.com>.
This sounds like a very reasonable idea except that the client contract
currently guarantees serialized execution of watcher callbacks. This can be
very hard to do without.
Having this be configurable with a strong default for the current behavior
might be nice, though.
Do you have a suggested patch?
On Mon, Aug 15, 2011 at 2:15 PM, Martin Serrano <ma...@attivio.com> wrote:
> Hi,
>
> Has the zookeeper community considered adding an option to use a thread
> pool executor for event dispatch? One pattern we have developed for our
> library is to trigger event listeners in a separate thread to guard against
> blocking operations in listeners. A blocking watcher will prevent new
> events/watchers from being triggered. By adding the ability to configure
> which Executor was used by a zookeeper client for event delivery library
> writers could make the choice that is best for them. Being able to do this
> at the client level (rather than for each individual watcher class I
> implement) is a cleaner approach IMO.
>
> Thoughts?
>
> -Martin
>