You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Rich Schumacher <ri...@gmail.com> on 2009/12/16 01:38:06 UTC

Return data size in zkpython

Hey all,

I'm working on using ZooKeeper for an internal application at Digg.  I've been using the zkpython package and I just noticed that the data I was receiving from a zookeeper.get() call was being truncated.  After some quick digging I found that zookeeper.c limits the data returned to 512 characters (see http://svn.apache.org/viewvc/hadoop/zookeeper/tags/release-3.2.2/src/contrib/zkpython/src/c/zookeeper.c?view=markup line 855).

Is there a reason for this?  The only information regarding node size that I've read is that it should not exceed 1MB so this limit seems a bit arbitrary and restrictive.

Thanks for the great work!

Rich

Re: Return data size in zkpython

Posted by Rich Schumacher <ri...@gmail.com>.
Hey Henry,

Thanks for the quick response and patch.  I was going to cobble together a quick patch but this is a much more robust solution.  I'll implement this fix and let you know if I run into any problems with it.

Thanks again!

Rich

On Dec 15, 2009, at 6:41 PM, Henry Robinson wrote:

> Hi -
> 
> See https://issues.apache.org/jira/browse/ZOOKEEPER-627, and the attached
> patch. I've upped the limit to a 1Mb buffer. Also I've added a fourth
> parameter to zookeeper.get - if you set this integer parameter to the size
> of the buffer you are expecting, zkpython will return no more than this many
> bytes.
> 
> Thanks again for flagging this up.
> 
> cheers,
> Henry
> 
> On Tue, Dec 15, 2009 at 4:43 PM, Henry Robinson <he...@cloudera.com> wrote:
> 
>> Hey Rich -
>> 
>> That's a really dumb restriction :) I'll open a JIRA and get it fixed asap.
>> 
>> Thanks for the report!
>> 
>> Henry
>> 
>> 
>> On Tue, Dec 15, 2009 at 4:38 PM, Rich Schumacher <ri...@gmail.com>wrote:
>> 
>>> Hey all,
>>> 
>>> I'm working on using ZooKeeper for an internal application at Digg.  I've
>>> been using the zkpython package and I just noticed that the data I was
>>> receiving from a zookeeper.get() call was being truncated.  After some quick
>>> digging I found that zookeeper.c limits the data returned to 512 characters
>>> (see
>>> http://svn.apache.org/viewvc/hadoop/zookeeper/tags/release-3.2.2/src/contrib/zkpython/src/c/zookeeper.c?view=markupline 855).
>>> 
>>> Is there a reason for this?  The only information regarding node size that
>>> I've read is that it should not exceed 1MB so this limit seems a bit
>>> arbitrary and restrictive.
>>> 
>>> Thanks for the great work!
>>> 
>>> Rich
>> 
>> 
>>

Re: Return data size in zkpython

Posted by Henry Robinson <he...@cloudera.com>.
Hi -

See https://issues.apache.org/jira/browse/ZOOKEEPER-627, and the attached
patch. I've upped the limit to a 1Mb buffer. Also I've added a fourth
parameter to zookeeper.get - if you set this integer parameter to the size
of the buffer you are expecting, zkpython will return no more than this many
bytes.

Thanks again for flagging this up.

cheers,
Henry

On Tue, Dec 15, 2009 at 4:43 PM, Henry Robinson <he...@cloudera.com> wrote:

> Hey Rich -
>
> That's a really dumb restriction :) I'll open a JIRA and get it fixed asap.
>
> Thanks for the report!
>
> Henry
>
>
> On Tue, Dec 15, 2009 at 4:38 PM, Rich Schumacher <ri...@gmail.com>wrote:
>
>> Hey all,
>>
>> I'm working on using ZooKeeper for an internal application at Digg.  I've
>> been using the zkpython package and I just noticed that the data I was
>> receiving from a zookeeper.get() call was being truncated.  After some quick
>> digging I found that zookeeper.c limits the data returned to 512 characters
>> (see
>> http://svn.apache.org/viewvc/hadoop/zookeeper/tags/release-3.2.2/src/contrib/zkpython/src/c/zookeeper.c?view=markupline 855).
>>
>> Is there a reason for this?  The only information regarding node size that
>> I've read is that it should not exceed 1MB so this limit seems a bit
>> arbitrary and restrictive.
>>
>> Thanks for the great work!
>>
>> Rich
>
>
>

Re: Return data size in zkpython

Posted by Henry Robinson <he...@cloudera.com>.
Hey Rich -

That's a really dumb restriction :) I'll open a JIRA and get it fixed asap.

Thanks for the report!

Henry

On Tue, Dec 15, 2009 at 4:38 PM, Rich Schumacher <ri...@gmail.com>wrote:

> Hey all,
>
> I'm working on using ZooKeeper for an internal application at Digg.  I've
> been using the zkpython package and I just noticed that the data I was
> receiving from a zookeeper.get() call was being truncated.  After some quick
> digging I found that zookeeper.c limits the data returned to 512 characters
> (see
> http://svn.apache.org/viewvc/hadoop/zookeeper/tags/release-3.2.2/src/contrib/zkpython/src/c/zookeeper.c?view=markupline 855).
>
> Is there a reason for this?  The only information regarding node size that
> I've read is that it should not exceed 1MB so this limit seems a bit
> arbitrary and restrictive.
>
> Thanks for the great work!
>
> Rich