You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by saurabh jain <sa...@gmail.com> on 2014/08/15 08:41:22 UTC
Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Hi Folks,
I am planning to use ZooKeeper znodes to store my key value data.
Name of the znode will be my key and data present inside it will be my
value.
The problem is I can have millions of key/value pairs.
Is the zookeeper recommended to solve this type of problem. I read about
the zookeeper that
it is not an actual File system but a splitted File system and should be
used only for distributed coordination service.
My requirement is something like this that I need a global place to store
these key values so that all the jobs can access it and even if some job
create a new znode then my other jobs can see these changes with the help
of watcher.
Would you guys recommend using ZooKeeper for above problem statement ?
I have also read in one of the mail archive on the limitation of max no of
znodes , if this limitation still exists then may be this solution won't
work in my case.
http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
Please advise
Many Thanks
Saurabh
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Rakesh Radhakrishnan <ra...@gmail.com>.
BK uses ZooKeeper :-
1) to store its metadata information about the ledgers(you can assume
'ledger' as a 'file' in the filesystem).
2) to store the details about the available Bookie servers (like service
discovery)
Basic terms are like:-
- servers called as"bookies",
- file are "ledgers",
- and each unit of a file (aka record) is a "ledger entry"
On Sat, Aug 16, 2014 at 12:40 AM, Alvaro Gareppe <ag...@gmail.com> wrote:
> nice to know.. BookKeeper is build in top of zookeeper ?
>
>
> On Fri, Aug 15, 2014 at 4:08 PM, Rakesh Radhakrishnan <
> rakeshr.apache@gmail.com> wrote:
>
> > Adding one more, you can also explore Apache BookKeeper for storing
> > millions
> > of K/V data. Since it uses filesystem for storing the ledgers & its
> > entries(data) size won't be a constraint. But it doesn't have watch
> > notifications.
> >
> > Regards,
> > Rakesh
> >
> >
> >
> >
> > On Fri, Aug 15, 2014 at 10:18 PM, Alvaro Gareppe <ag...@gmail.com>
> > wrote:
> >
> > > It doesn't seems to be the best use of zookeeper, Maybe redis is a best
> > > "tool" for you.. or some no sql db like Casandra or mongo is going to
> > fit a
> > > little bit more.
> > >
> > >
> > >
> > >
> > > On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Folks,
> > > >
> > > > I am planning to use ZooKeeper znodes to store my key value data.
> > > > Name of the znode will be my key and data present inside it will be
> my
> > > > value.
> > > >
> > > >
> > > > The problem is I can have millions of key/value pairs.
> > > >
> > > > Is the zookeeper recommended to solve this type of problem. I read
> > about
> > > > the zookeeper that
> > > > it is not an actual File system but a splitted File system and
> should
> > be
> > > > used only for distributed coordination service.
> > > >
> > > > My requirement is something like this that I need a global place to
> > store
> > > > these key values so that all the jobs can access it and even if some
> > job
> > > > create a new znode then my other jobs can see these changes with the
> > help
> > > > of watcher.
> > > >
> > > > Would you guys recommend using ZooKeeper for above problem statement
> ?
> > > >
> > > > I have also read in one of the mail archive on the limitation of max
> no
> > > of
> > > > znodes , if this limitation still exists then may be this solution
> > won't
> > > > work in my case.
> > > >
> > > >
> > > >
> > >
> >
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
> > > >
> > > > Please advise
> > > >
> > > > Many Thanks
> > > > Saurabh
> > > >
> > >
> > >
> > >
> > > --
> > > Ing. Alvaro Gareppe
> > > agareppe@gmail.com
> > >
> >
>
>
>
> --
> Ing. Alvaro Gareppe
> agareppe@gmail.com
>
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Rakesh Radhakrishnan <ra...@gmail.com>.
BK uses ZooKeeper :-
1) to store its metadata information about the ledgers(you can assume
'ledger' as a 'file' in the filesystem).
2) to store the details about the available Bookie servers (like service
discovery)
Basic terms are like:-
- servers called as"bookies",
- file are "ledgers",
- and each unit of a file (aka record) is a "ledger entry"
On Sat, Aug 16, 2014 at 12:40 AM, Alvaro Gareppe <ag...@gmail.com> wrote:
> nice to know.. BookKeeper is build in top of zookeeper ?
>
>
> On Fri, Aug 15, 2014 at 4:08 PM, Rakesh Radhakrishnan <
> rakeshr.apache@gmail.com> wrote:
>
> > Adding one more, you can also explore Apache BookKeeper for storing
> > millions
> > of K/V data. Since it uses filesystem for storing the ledgers & its
> > entries(data) size won't be a constraint. But it doesn't have watch
> > notifications.
> >
> > Regards,
> > Rakesh
> >
> >
> >
> >
> > On Fri, Aug 15, 2014 at 10:18 PM, Alvaro Gareppe <ag...@gmail.com>
> > wrote:
> >
> > > It doesn't seems to be the best use of zookeeper, Maybe redis is a best
> > > "tool" for you.. or some no sql db like Casandra or mongo is going to
> > fit a
> > > little bit more.
> > >
> > >
> > >
> > >
> > > On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com>
> > > wrote:
> > >
> > > > Hi Folks,
> > > >
> > > > I am planning to use ZooKeeper znodes to store my key value data.
> > > > Name of the znode will be my key and data present inside it will be
> my
> > > > value.
> > > >
> > > >
> > > > The problem is I can have millions of key/value pairs.
> > > >
> > > > Is the zookeeper recommended to solve this type of problem. I read
> > about
> > > > the zookeeper that
> > > > it is not an actual File system but a splitted File system and
> should
> > be
> > > > used only for distributed coordination service.
> > > >
> > > > My requirement is something like this that I need a global place to
> > store
> > > > these key values so that all the jobs can access it and even if some
> > job
> > > > create a new znode then my other jobs can see these changes with the
> > help
> > > > of watcher.
> > > >
> > > > Would you guys recommend using ZooKeeper for above problem statement
> ?
> > > >
> > > > I have also read in one of the mail archive on the limitation of max
> no
> > > of
> > > > znodes , if this limitation still exists then may be this solution
> > won't
> > > > work in my case.
> > > >
> > > >
> > > >
> > >
> >
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
> > > >
> > > > Please advise
> > > >
> > > > Many Thanks
> > > > Saurabh
> > > >
> > >
> > >
> > >
> > > --
> > > Ing. Alvaro Gareppe
> > > agareppe@gmail.com
> > >
> >
>
>
>
> --
> Ing. Alvaro Gareppe
> agareppe@gmail.com
>
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Alvaro Gareppe <ag...@gmail.com>.
nice to know.. BookKeeper is build in top of zookeeper ?
On Fri, Aug 15, 2014 at 4:08 PM, Rakesh Radhakrishnan <
rakeshr.apache@gmail.com> wrote:
> Adding one more, you can also explore Apache BookKeeper for storing
> millions
> of K/V data. Since it uses filesystem for storing the ledgers & its
> entries(data) size won't be a constraint. But it doesn't have watch
> notifications.
>
> Regards,
> Rakesh
>
>
>
>
> On Fri, Aug 15, 2014 at 10:18 PM, Alvaro Gareppe <ag...@gmail.com>
> wrote:
>
> > It doesn't seems to be the best use of zookeeper, Maybe redis is a best
> > "tool" for you.. or some no sql db like Casandra or mongo is going to
> fit a
> > little bit more.
> >
> >
> >
> >
> > On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com>
> > wrote:
> >
> > > Hi Folks,
> > >
> > > I am planning to use ZooKeeper znodes to store my key value data.
> > > Name of the znode will be my key and data present inside it will be my
> > > value.
> > >
> > >
> > > The problem is I can have millions of key/value pairs.
> > >
> > > Is the zookeeper recommended to solve this type of problem. I read
> about
> > > the zookeeper that
> > > it is not an actual File system but a splitted File system and should
> be
> > > used only for distributed coordination service.
> > >
> > > My requirement is something like this that I need a global place to
> store
> > > these key values so that all the jobs can access it and even if some
> job
> > > create a new znode then my other jobs can see these changes with the
> help
> > > of watcher.
> > >
> > > Would you guys recommend using ZooKeeper for above problem statement ?
> > >
> > > I have also read in one of the mail archive on the limitation of max no
> > of
> > > znodes , if this limitation still exists then may be this solution
> won't
> > > work in my case.
> > >
> > >
> > >
> >
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
> > >
> > > Please advise
> > >
> > > Many Thanks
> > > Saurabh
> > >
> >
> >
> >
> > --
> > Ing. Alvaro Gareppe
> > agareppe@gmail.com
> >
>
--
Ing. Alvaro Gareppe
agareppe@gmail.com
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Alvaro Gareppe <ag...@gmail.com>.
nice to know.. BookKeeper is build in top of zookeeper ?
On Fri, Aug 15, 2014 at 4:08 PM, Rakesh Radhakrishnan <
rakeshr.apache@gmail.com> wrote:
> Adding one more, you can also explore Apache BookKeeper for storing
> millions
> of K/V data. Since it uses filesystem for storing the ledgers & its
> entries(data) size won't be a constraint. But it doesn't have watch
> notifications.
>
> Regards,
> Rakesh
>
>
>
>
> On Fri, Aug 15, 2014 at 10:18 PM, Alvaro Gareppe <ag...@gmail.com>
> wrote:
>
> > It doesn't seems to be the best use of zookeeper, Maybe redis is a best
> > "tool" for you.. or some no sql db like Casandra or mongo is going to
> fit a
> > little bit more.
> >
> >
> >
> >
> > On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com>
> > wrote:
> >
> > > Hi Folks,
> > >
> > > I am planning to use ZooKeeper znodes to store my key value data.
> > > Name of the znode will be my key and data present inside it will be my
> > > value.
> > >
> > >
> > > The problem is I can have millions of key/value pairs.
> > >
> > > Is the zookeeper recommended to solve this type of problem. I read
> about
> > > the zookeeper that
> > > it is not an actual File system but a splitted File system and should
> be
> > > used only for distributed coordination service.
> > >
> > > My requirement is something like this that I need a global place to
> store
> > > these key values so that all the jobs can access it and even if some
> job
> > > create a new znode then my other jobs can see these changes with the
> help
> > > of watcher.
> > >
> > > Would you guys recommend using ZooKeeper for above problem statement ?
> > >
> > > I have also read in one of the mail archive on the limitation of max no
> > of
> > > znodes , if this limitation still exists then may be this solution
> won't
> > > work in my case.
> > >
> > >
> > >
> >
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
> > >
> > > Please advise
> > >
> > > Many Thanks
> > > Saurabh
> > >
> >
> >
> >
> > --
> > Ing. Alvaro Gareppe
> > agareppe@gmail.com
> >
>
--
Ing. Alvaro Gareppe
agareppe@gmail.com
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Rakesh Radhakrishnan <ra...@gmail.com>.
Adding one more, you can also explore Apache BookKeeper for storing millions
of K/V data. Since it uses filesystem for storing the ledgers & its
entries(data) size won't be a constraint. But it doesn't have watch
notifications.
Regards,
Rakesh
On Fri, Aug 15, 2014 at 10:18 PM, Alvaro Gareppe <ag...@gmail.com> wrote:
> It doesn't seems to be the best use of zookeeper, Maybe redis is a best
> "tool" for you.. or some no sql db like Casandra or mongo is going to fit a
> little bit more.
>
>
>
>
> On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > I am planning to use ZooKeeper znodes to store my key value data.
> > Name of the znode will be my key and data present inside it will be my
> > value.
> >
> >
> > The problem is I can have millions of key/value pairs.
> >
> > Is the zookeeper recommended to solve this type of problem. I read about
> > the zookeeper that
> > it is not an actual File system but a splitted File system and should be
> > used only for distributed coordination service.
> >
> > My requirement is something like this that I need a global place to store
> > these key values so that all the jobs can access it and even if some job
> > create a new znode then my other jobs can see these changes with the help
> > of watcher.
> >
> > Would you guys recommend using ZooKeeper for above problem statement ?
> >
> > I have also read in one of the mail archive on the limitation of max no
> of
> > znodes , if this limitation still exists then may be this solution won't
> > work in my case.
> >
> >
> >
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
> >
> > Please advise
> >
> > Many Thanks
> > Saurabh
> >
>
>
>
> --
> Ing. Alvaro Gareppe
> agareppe@gmail.com
>
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Rakesh Radhakrishnan <ra...@gmail.com>.
Adding one more, you can also explore Apache BookKeeper for storing millions
of K/V data. Since it uses filesystem for storing the ledgers & its
entries(data) size won't be a constraint. But it doesn't have watch
notifications.
Regards,
Rakesh
On Fri, Aug 15, 2014 at 10:18 PM, Alvaro Gareppe <ag...@gmail.com> wrote:
> It doesn't seems to be the best use of zookeeper, Maybe redis is a best
> "tool" for you.. or some no sql db like Casandra or mongo is going to fit a
> little bit more.
>
>
>
>
> On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > I am planning to use ZooKeeper znodes to store my key value data.
> > Name of the znode will be my key and data present inside it will be my
> > value.
> >
> >
> > The problem is I can have millions of key/value pairs.
> >
> > Is the zookeeper recommended to solve this type of problem. I read about
> > the zookeeper that
> > it is not an actual File system but a splitted File system and should be
> > used only for distributed coordination service.
> >
> > My requirement is something like this that I need a global place to store
> > these key values so that all the jobs can access it and even if some job
> > create a new znode then my other jobs can see these changes with the help
> > of watcher.
> >
> > Would you guys recommend using ZooKeeper for above problem statement ?
> >
> > I have also read in one of the mail archive on the limitation of max no
> of
> > znodes , if this limitation still exists then may be this solution won't
> > work in my case.
> >
> >
> >
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
> >
> > Please advise
> >
> > Many Thanks
> > Saurabh
> >
>
>
>
> --
> Ing. Alvaro Gareppe
> agareppe@gmail.com
>
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Alvaro Gareppe <ag...@gmail.com>.
It doesn't seems to be the best use of zookeeper, Maybe redis is a best
"tool" for you.. or some no sql db like Casandra or mongo is going to fit a
little bit more.
On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com> wrote:
> Hi Folks,
>
> I am planning to use ZooKeeper znodes to store my key value data.
> Name of the znode will be my key and data present inside it will be my
> value.
>
>
> The problem is I can have millions of key/value pairs.
>
> Is the zookeeper recommended to solve this type of problem. I read about
> the zookeeper that
> it is not an actual File system but a splitted File system and should be
> used only for distributed coordination service.
>
> My requirement is something like this that I need a global place to store
> these key values so that all the jobs can access it and even if some job
> create a new znode then my other jobs can see these changes with the help
> of watcher.
>
> Would you guys recommend using ZooKeeper for above problem statement ?
>
> I have also read in one of the mail archive on the limitation of max no of
> znodes , if this limitation still exists then may be this solution won't
> work in my case.
>
>
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
>
> Please advise
>
> Many Thanks
> Saurabh
>
--
Ing. Alvaro Gareppe
agareppe@gmail.com
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Alvaro Gareppe <ag...@gmail.com>.
It doesn't seems to be the best use of zookeeper, Maybe redis is a best
"tool" for you.. or some no sql db like Casandra or mongo is going to fit a
little bit more.
On Fri, Aug 15, 2014 at 3:41 AM, saurabh jain <sa...@gmail.com> wrote:
> Hi Folks,
>
> I am planning to use ZooKeeper znodes to store my key value data.
> Name of the znode will be my key and data present inside it will be my
> value.
>
>
> The problem is I can have millions of key/value pairs.
>
> Is the zookeeper recommended to solve this type of problem. I read about
> the zookeeper that
> it is not an actual File system but a splitted File system and should be
> used only for distributed coordination service.
>
> My requirement is something like this that I need a global place to store
> these key values so that all the jobs can access it and even if some job
> create a new znode then my other jobs can see these changes with the help
> of watcher.
>
> Would you guys recommend using ZooKeeper for above problem statement ?
>
> I have also read in one of the mail archive on the limitation of max no of
> znodes , if this limitation still exists then may be this solution won't
> work in my case.
>
>
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
>
> Please advise
>
> Many Thanks
> Saurabh
>
--
Ing. Alvaro Gareppe
agareppe@gmail.com
Re: Zookeeper for storing huge centralize data and Max possible No of
zNodes - Please advise
Posted by Sékine Coulibaly <sc...@gmail.com>.
I've no experience of such a deployment. Depending on your read/write
access pattern, this might be a fit for Zookeeper. However, if you
intensively use watches, I'd say the performance will be hit.
Do you use Zookeeper for other purpose than your K/V store ? Why don't you
stick to a distributed database instead ? Some of them do offer watches
such as Redis which offers keyspace notifications (beware, in-memory).
Others will need to be hacked since AFAIK they rarely offer watches
(MongoDb, Cassandra...). This will do the job,
Maybe others will have others ideas.
2014-08-15 8:41 GMT+02:00 saurabh jain <sa...@gmail.com>:
> Hi Folks,
>
> I am planning to use ZooKeeper znodes to store my key value data.
> Name of the znode will be my key and data present inside it will be my
> value.
>
>
> The problem is I can have millions of key/value pairs.
>
> Is the zookeeper recommended to solve this type of problem. I read about
> the zookeeper that
> it is not an actual File system but a splitted File system and should be
> used only for distributed coordination service.
>
> My requirement is something like this that I need a global place to store
> these key values so that all the jobs can access it and even if some job
> create a new znode then my other jobs can see these changes with the help
> of watcher.
>
> Would you guys recommend using ZooKeeper for above problem statement ?
>
> I have also read in one of the mail archive on the limitation of max no of
> znodes , if this limitation still exists then may be this solution won't
> work in my case.
>
>
> http://zookeeper-user.578899.n2.nabble.com/Question-regarding-the-maximum-number-of-ZNODES-a-zookeeper-td6979604.html
>
> Please advise
>
> Many Thanks
> Saurabh
>