zkpython improvements

[Duncan Findlay <du...@duncf.ca>]

I've been working on some improvements to the ZooKeeper python bindings and I was hoping (eventually) to submit them upstream for inclusion in ZooKeeper.

Most of my work so far has been around making the synchronous API calls available through a pure-Python object-oriented interface instead of needing to keep track and pass zk client ids around everywhere, and cleaning up some of calling conventions. At this point I haven't implemented the asynchronous API, mostly because I don't use it, and I'm not sure what the best way to handle callbacks in a clean way.

My work is available here:
https://github.com/duncf/zkpython

Before I open a JIRA ticket with a patch, I was hoping to get some consensus around the API and get some understanding around the following points:

- How important is backwards compatibility for things like zkpython between versions?

- Do we need support for asynchronous methods before considering changes for inclusion in ZooKeeper source? (Would anybody like to help?)

Thanks
Duncan

Re: zkpython improvements

[Duncan Findlay <du...@duncf.ca>]

On Apr 4, 2012, at 4:19 PM, Henry Robinson wrote:
>> My work is available here:
>> https://github.com/duncf/zkpython
>> 
>> Before I open a JIRA ticket with a patch, I was hoping to get some
>> consensus around the API and get some understanding around the following
>> points:
>> 
>> - How important is backwards compatibility for things like zkpython
>> between versions?
>> 
> 
> Unfortunately I think it's quite important to maintain for the 3.x series
> of ZK. That said, if there's a way we can keep both APIs around there's no
> problem adding your improved API to the original one (as long as everyone
> agrees it's the right direction for the API to go) - that'll help with the
> transition. How feasible do you think that might be? I haven't read your
> patch series yet, so I don't know how much of the guts has been rewritten,
> but it looks as though zookeeper.c survived fairly intact so perhaps both
> APIs can coexist.

It should be fairly trivial to maintain backwards compatibility (just need to add a "from _zookeeper import *" or something equivalent). Knowing that we need to maintain backwards compatibility also puts on a certain amount of pressure to get it right. ;-)

>> 
>> - Do we need support for asynchronous methods before considering changes
>> for inclusion in ZooKeeper source? (Would anybody like to help?)
>> 
> 
> I think async support is required if this ever gets into a release and the
> old API doesn't, otherwise that's a functionality regression I'd like to
> avoid.

Understood. I've been brainstorming some ideas on how best to implement the async functionality. One of the difficulties (in my mind, at least) with the async API is that there are too many different types of callbacks, and remembering the callback arguments and their order. I've been thinking about making the callbacks in zkpython take an "Event" object which would have the appropriate attributes set. This means you could share callbacks for different types of calls. (You could potentially also share code between watchers and callbacks, but I'm not sure if that's desirable.)

I'm still pretty new to ZooKeeper, so I'm not sure about all the use cases; I'd love to hear some feedback on this.

Thanks
Duncan

Re: zkpython improvements

[Henry Robinson <he...@cloudera.com>]

Hi Duncan -

Great that you're working on the python bindings! That's very cool.

Regarding your questions, answers inline.

On 28 March 2012 14:52, Duncan Findlay <du...@duncf.ca> wrote:

> I've been working on some improvements to the ZooKeeper python bindings
> and I was hoping (eventually) to submit them upstream for inclusion in
> ZooKeeper.
>
> Most of my work so far has been around making the synchronous API calls
> available through a pure-Python object-oriented interface instead of
> needing to keep track and pass zk client ids around everywhere, and
> cleaning up some of calling conventions. At this point I haven't
> implemented the asynchronous API, mostly because I don't use it, and I'm
> not sure what the best way to handle callbacks in a clean way.


> My work is available here:
> https://github.com/duncf/zkpython
>
> Before I open a JIRA ticket with a patch, I was hoping to get some
> consensus around the API and get some understanding around the following
> points:
>
> - How important is backwards compatibility for things like zkpython
> between versions?
>

Unfortunately I think it's quite important to maintain for the 3.x series
of ZK. That said, if there's a way we can keep both APIs around there's no
problem adding your improved API to the original one (as long as everyone
agrees it's the right direction for the API to go) - that'll help with the
transition. How feasible do you think that might be? I haven't read your
patch series yet, so I don't know how much of the guts has been rewritten,
but it looks as though zookeeper.c survived fairly intact so perhaps both
APIs can coexist.



>
> - Do we need support for asynchronous methods before considering changes
> for inclusion in ZooKeeper source? (Would anybody like to help?)
>

I think async support is required if this ever gets into a release and the
old API doesn't, otherwise that's a functionality regression I'd like to
avoid.

Henry



>
> Thanks
> Duncan




-- 
Henry Robinson
Software Engineer
Cloudera
415-994-6679