You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Will Johnson <wi...@gmail.com> on 2011/08/11 15:08:28 UTC
raise 1m max node data size
We have a situation where 99.9% of all data stored in zookeeper will be well
under the 1mb limit (probably under 1k as well) but there is a small
possibility that at some point users may do something to cross that
barrier. I'd like to raise the max to a higher number realizing that if we
do hit that case, performance may suffer but it's better than having the app
crash. i've looked through the docs and code and tried changing the
org.apache.zookeeper.server.quorum.QuorumCnxManager.PACKETMAXSIZE to a
larger number but something still seems to be blocking my test of larger
data sizes.
Is there some configuration parameter I am missing or code change i can
make? Or have people solved this another way? My first inclination was to
split up larger data streams across multiple nodes but that seems to cause
lots of problems with watches and atomicity that i don't think are easily
solvable.
- will
Re: raise 1m max node data size
Posted by Jared Cantwell <ja...@gmail.com>.
I think you can achieve this by using the command line argument to the
server:
-Djute.maxbuffer=<bytes>
~Jared
On Thu, Aug 11, 2011 at 7:08 AM, Will Johnson
<wi...@gmail.com>wrote:
> We have a situation where 99.9% of all data stored in zookeeper will be
> well
> under the 1mb limit (probably under 1k as well) but there is a small
> possibility that at some point users may do something to cross that
> barrier. I'd like to raise the max to a higher number realizing that if we
> do hit that case, performance may suffer but it's better than having the
> app
> crash. i've looked through the docs and code and tried changing the
> org.apache.zookeeper.server.quorum.QuorumCnxManager.PACKETMAXSIZE to a
> larger number but something still seems to be blocking my test of larger
> data sizes.
>
> Is there some configuration parameter I am missing or code change i can
> make? Or have people solved this another way? My first inclination was to
> split up larger data streams across multiple nodes but that seems to cause
> lots of problems with watches and atomicity that i don't think are easily
> solvable.
>
> - will
>
Re: raise 1m max node data size
Posted by Jared Cantwell <ja...@gmail.com>.
Ahh, I missed that in your message. This is a clever use of multi.
~Jared
On Thu, Aug 11, 2011 at 2:16 PM, Ted Dunning <te...@gmail.com> wrote:
> Yes.
>
> But you can commit a single coordination file with version checks on 5
> pre-existing 1MB files.
>
> On Thu, Aug 11, 2011 at 11:00 AM, Jared Cantwell
> <ja...@gmail.com>wrote:
>
> > Currently, the limit on the sum of the sizes of the updates in the multi
> > command is still 1MB. You cannot commit 5 1MB nodes in a multi-op.
> >
> > ~Jared
> >
> > On Thu, Aug 11, 2011 at 11:20 AM, Ted Dunning <te...@gmail.com>
> > wrote:
> >
> > > Another way to solve this is to use the multi command.
> > >
> > > The idea would be that you would upload multiple pieces of the large
> > object
> > > separately into different znodes (without using multi)
> > >
> > > Then you would update a pointer node that has references to the pieces
> > > while
> > > controlling for the version of the pieces (using a multi).
> > >
> > > On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
> > > <wi...@gmail.com>wrote:
> > >
> > > > We have a situation where 99.9% of all data stored in zookeeper will
> be
> > > > well
> > > > under the 1mb limit (probably under 1k as well) but there is a small
> > > > possibility that at some point users may do something to cross that
> > > > barrier. ...
> > > > Is there some configuration parameter I am missing or code change i
> can
> > > > make? Or have people solved this another way?
> > > >
> > >
> >
>
Re: raise 1m max node data size
Posted by Ted Dunning <te...@gmail.com>.
Yes.
But you can commit a single coordination file with version checks on 5
pre-existing 1MB files.
On Thu, Aug 11, 2011 at 11:00 AM, Jared Cantwell
<ja...@gmail.com>wrote:
> Currently, the limit on the sum of the sizes of the updates in the multi
> command is still 1MB. You cannot commit 5 1MB nodes in a multi-op.
>
> ~Jared
>
> On Thu, Aug 11, 2011 at 11:20 AM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > Another way to solve this is to use the multi command.
> >
> > The idea would be that you would upload multiple pieces of the large
> object
> > separately into different znodes (without using multi)
> >
> > Then you would update a pointer node that has references to the pieces
> > while
> > controlling for the version of the pieces (using a multi).
> >
> > On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
> > <wi...@gmail.com>wrote:
> >
> > > We have a situation where 99.9% of all data stored in zookeeper will be
> > > well
> > > under the 1mb limit (probably under 1k as well) but there is a small
> > > possibility that at some point users may do something to cross that
> > > barrier. ...
> > > Is there some configuration parameter I am missing or code change i can
> > > make? Or have people solved this another way?
> > >
> >
>
Re: raise 1m max node data size
Posted by Jared Cantwell <ja...@gmail.com>.
Currently, the limit on the sum of the sizes of the updates in the multi
command is still 1MB. You cannot commit 5 1MB nodes in a multi-op.
~Jared
On Thu, Aug 11, 2011 at 11:20 AM, Ted Dunning <te...@gmail.com> wrote:
> Another way to solve this is to use the multi command.
>
> The idea would be that you would upload multiple pieces of the large object
> separately into different znodes (without using multi)
>
> Then you would update a pointer node that has references to the pieces
> while
> controlling for the version of the pieces (using a multi).
>
> On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
> <wi...@gmail.com>wrote:
>
> > We have a situation where 99.9% of all data stored in zookeeper will be
> > well
> > under the 1mb limit (probably under 1k as well) but there is a small
> > possibility that at some point users may do something to cross that
> > barrier. ...
> > Is there some configuration parameter I am missing or code change i can
> > make? Or have people solved this another way?
> >
>
Re: raise 1m max node data size
Posted by Ted Dunning <te...@gmail.com>.
Another way to solve this is to use the multi command.
The idea would be that you would upload multiple pieces of the large object
separately into different znodes (without using multi)
Then you would update a pointer node that has references to the pieces while
controlling for the version of the pieces (using a multi).
On Thu, Aug 11, 2011 at 6:08 AM, Will Johnson
<wi...@gmail.com>wrote:
> We have a situation where 99.9% of all data stored in zookeeper will be
> well
> under the 1mb limit (probably under 1k as well) but there is a small
> possibility that at some point users may do something to cross that
> barrier. ...
> Is there some configuration parameter I am missing or code change i can
> make? Or have people solved this another way?
>