Running Zookeeper in clustered setup/

[lalit jangra <la...@gmail.com>]

Hi,

I am a newbie in Zookeeper are so please apologize for any not so relevant
question asked.

I am running clustered Apache ManifoldCF on two nodes which are pointing to
same database. But i need synchronization between both nodes so avoid race
condition while both nodes are working together.

For same, as per Apache ManifoldCF guidelines, i am using Zookeeper to
manage synchronization in cluster.Also as per ManifoldCF & Zookeeper
instructions, i have setup & configured two set of three Zookeeper server
at both nodes which are working fine.

But i have a clustered setup where both ManifoldCF nodes point to same DB ,
i am still clueless about how should i proceed to connect Zookeeper nodes
at both servers to make a single setup.

Also i am not sure whether it is necessary or not but then the question
arises that how two servers & Zookeeper nodes in cluster will talk to each
other?

So normally what should be the steps taken once zookeeper servers are setup
to maintain synchronization.

Regards,
Lalit Jangra.

Re: Running Zookeeper in clustered setup/

[Cameron McKenzie <mc...@gmail.com>]

Glad you got it going. I'm not much help with tuning, but the user guide
has a bit on it. I'm sure some other people on the list will be able to
help you if you have specific questions.
cheers


On Fri, Jul 4, 2014 at 8:29 PM, lalit jangra <la...@gmail.com>
wrote:

> Thanks Cam,
>
> It was very helpful and i have setup a cluster of 6 nodes on 2 nodes, with
> each node having 3 zookeeper servers and they all talk to each other and i
> could see that.
>
> I could also see that each zookeeper server creates a lots of data and logs
> which are related to transaction logs and snapshots to maintain high
> availability in the system. Currently i have configured everything in nodes
> only but i have read that data & logs should be in separate partition.
>
> Can you help on this? Also are there any more performance tuning guidelines
> available?
>
> Regards.
>
>
> On Fri, Jul 4, 2014 at 12:58 AM, Cameron McKenzie <mc...@gmail.com>
> wrote:
>
> > If you're running across two hosts (call them serverA and serverB) in a
> 2+1
> > configuration then your config should be.
> >
> > server.1=serverA:2888:3888
> > server.2=serverA:2889:3889
> > server.3=serverB:2888:3888
> >
> > Then in the configured data directory for each ZooKeeper node you need to
> > set up the myid file appropriately (i.e. It should contain 1 for
> server1, 2
> > for server2, 3 for server 3).
> >
> > cheers
> >
> >
> >
> > On Fri, Jul 4, 2014 at 8:02 AM, lalit jangra <la...@gmail.com>
> > wrote:
> >
> > > Thanks Cam,
> > >
> > > So i should have either 2+1 or 3+2 Zookeeper servers on these tow nodes
> > to
> > > which both nodes are talking?
> > >
> > > In this case, what should i put in zoo.cfg file for host values as
> > > currently i am using localhost at both nodes.
> > >
> > > server.1=localhost:2888:3888
> > > server.1=localhost:2889:3889
> > > server.1=localhost:28908:3890
> > >
> > > Again how these 3 nodes will form a cluster?
> > >
> > > Regards.
> > >
> > >
> > > On Thu, Jul 3, 2014 at 10:27 PM, Cameron McKenzie <
> > mckenzie.cam@gmail.com>
> > > wrote:
> > >
> > > > hey Lalit,
> > > > I'm not familiar with Apache ManifoldCF, but the configuration
> sounds a
> > > bit
> > > > strange. You've essentially got two independent ZooKeeper clusters
> set
> > > up.
> > > > They won't communicate with each other at all. I'd presume that if
> > you've
> > > > got a database shared between your ManifoldCF instances, that you'd
> > also
> > > > want your ZooKeeper cluster to be shared between these two instances.
> > > >
> > > > In this case, you'd normally run your ZooKeeper nodes across both
> sites
> > > (2
> > > > + 1, or 3 + 2) and then have both of your client (ManifoldCF) nodes
> > > > communicate with this cluster.
> > > >
> > > > Having said that, there may be a reason that ManifoldCF uses two
> > > > independent clusters that don't communicate with each other, I'm not
> > > sure.
> > > >
> > > > cheers
> > > > Cam
> > > >
> > > >
> > > >
> > > > On Fri, Jul 4, 2014 at 7:17 AM, lalit jangra <
> lalit.j.jangra@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am a newbie in Zookeeper are so please apologize for any not so
> > > > relevant
> > > > > question asked.
> > > > >
> > > > > I am running clustered Apache ManifoldCF on two nodes which are
> > > pointing
> > > > to
> > > > > same database. But i need synchronization between both nodes so
> avoid
> > > > race
> > > > > condition while both nodes are working together.
> > > > >
> > > > > For same, as per Apache ManifoldCF guidelines, i am using Zookeeper
> > to
> > > > > manage synchronization in cluster.Also as per ManifoldCF &
> Zookeeper
> > > > > instructions, i have setup & configured two set of three Zookeeper
> > > server
> > > > > at both nodes which are working fine.
> > > > >
> > > > > But i have a clustered setup where both ManifoldCF nodes point to
> > same
> > > > DB ,
> > > > > i am still clueless about how should i proceed to connect Zookeeper
> > > nodes
> > > > > at both servers to make a single setup.
> > > > >
> > > > > Also i am not sure whether it is necessary or not but then the
> > question
> > > > > arises that how two servers & Zookeeper nodes in cluster will talk
> to
> > > > each
> > > > > other?
> > > > >
> > > > > So normally what should be the steps taken once zookeeper servers
> are
> > > > setup
> > > > > to maintain synchronization.
> > > > >
> > > > > Regards,
> > > > > Lalit Jangra.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Lalit Jangra.
> > >
> >
>
>
>
> --
> Regards,
> Lalit Jangra.
>

Re: Running Zookeeper in clustered setup/

[lalit jangra <la...@gmail.com>]

Thanks Cam,

It was very helpful and i have setup a cluster of 6 nodes on 2 nodes, with
each node having 3 zookeeper servers and they all talk to each other and i
could see that.

I could also see that each zookeeper server creates a lots of data and logs
which are related to transaction logs and snapshots to maintain high
availability in the system. Currently i have configured everything in nodes
only but i have read that data & logs should be in separate partition.

Can you help on this? Also are there any more performance tuning guidelines
available?

Regards.


On Fri, Jul 4, 2014 at 12:58 AM, Cameron McKenzie <mc...@gmail.com>
wrote:

> If you're running across two hosts (call them serverA and serverB) in a 2+1
> configuration then your config should be.
>
> server.1=serverA:2888:3888
> server.2=serverA:2889:3889
> server.3=serverB:2888:3888
>
> Then in the configured data directory for each ZooKeeper node you need to
> set up the myid file appropriately (i.e. It should contain 1 for server1, 2
> for server2, 3 for server 3).
>
> cheers
>
>
>
> On Fri, Jul 4, 2014 at 8:02 AM, lalit jangra <la...@gmail.com>
> wrote:
>
> > Thanks Cam,
> >
> > So i should have either 2+1 or 3+2 Zookeeper servers on these tow nodes
> to
> > which both nodes are talking?
> >
> > In this case, what should i put in zoo.cfg file for host values as
> > currently i am using localhost at both nodes.
> >
> > server.1=localhost:2888:3888
> > server.1=localhost:2889:3889
> > server.1=localhost:28908:3890
> >
> > Again how these 3 nodes will form a cluster?
> >
> > Regards.
> >
> >
> > On Thu, Jul 3, 2014 at 10:27 PM, Cameron McKenzie <
> mckenzie.cam@gmail.com>
> > wrote:
> >
> > > hey Lalit,
> > > I'm not familiar with Apache ManifoldCF, but the configuration sounds a
> > bit
> > > strange. You've essentially got two independent ZooKeeper clusters set
> > up.
> > > They won't communicate with each other at all. I'd presume that if
> you've
> > > got a database shared between your ManifoldCF instances, that you'd
> also
> > > want your ZooKeeper cluster to be shared between these two instances.
> > >
> > > In this case, you'd normally run your ZooKeeper nodes across both sites
> > (2
> > > + 1, or 3 + 2) and then have both of your client (ManifoldCF) nodes
> > > communicate with this cluster.
> > >
> > > Having said that, there may be a reason that ManifoldCF uses two
> > > independent clusters that don't communicate with each other, I'm not
> > sure.
> > >
> > > cheers
> > > Cam
> > >
> > >
> > >
> > > On Fri, Jul 4, 2014 at 7:17 AM, lalit jangra <lalit.j.jangra@gmail.com
> >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am a newbie in Zookeeper are so please apologize for any not so
> > > relevant
> > > > question asked.
> > > >
> > > > I am running clustered Apache ManifoldCF on two nodes which are
> > pointing
> > > to
> > > > same database. But i need synchronization between both nodes so avoid
> > > race
> > > > condition while both nodes are working together.
> > > >
> > > > For same, as per Apache ManifoldCF guidelines, i am using Zookeeper
> to
> > > > manage synchronization in cluster.Also as per ManifoldCF & Zookeeper
> > > > instructions, i have setup & configured two set of three Zookeeper
> > server
> > > > at both nodes which are working fine.
> > > >
> > > > But i have a clustered setup where both ManifoldCF nodes point to
> same
> > > DB ,
> > > > i am still clueless about how should i proceed to connect Zookeeper
> > nodes
> > > > at both servers to make a single setup.
> > > >
> > > > Also i am not sure whether it is necessary or not but then the
> question
> > > > arises that how two servers & Zookeeper nodes in cluster will talk to
> > > each
> > > > other?
> > > >
> > > > So normally what should be the steps taken once zookeeper servers are
> > > setup
> > > > to maintain synchronization.
> > > >
> > > > Regards,
> > > > Lalit Jangra.
> > > >
> > >
> >
> >
> >
> > --
> > Regards,
> > Lalit Jangra.
> >
>



-- 
Regards,
Lalit Jangra.

Re: Running Zookeeper in clustered setup/

[Cameron McKenzie <mc...@gmail.com>]

If you're running across two hosts (call them serverA and serverB) in a 2+1
configuration then your config should be.

server.1=serverA:2888:3888
server.2=serverA:2889:3889
server.3=serverB:2888:3888

Then in the configured data directory for each ZooKeeper node you need to
set up the myid file appropriately (i.e. It should contain 1 for server1, 2
for server2, 3 for server 3).

cheers



On Fri, Jul 4, 2014 at 8:02 AM, lalit jangra <la...@gmail.com>
wrote:

> Thanks Cam,
>
> So i should have either 2+1 or 3+2 Zookeeper servers on these tow nodes to
> which both nodes are talking?
>
> In this case, what should i put in zoo.cfg file for host values as
> currently i am using localhost at both nodes.
>
> server.1=localhost:2888:3888
> server.1=localhost:2889:3889
> server.1=localhost:28908:3890
>
> Again how these 3 nodes will form a cluster?
>
> Regards.
>
>
> On Thu, Jul 3, 2014 at 10:27 PM, Cameron McKenzie <mc...@gmail.com>
> wrote:
>
> > hey Lalit,
> > I'm not familiar with Apache ManifoldCF, but the configuration sounds a
> bit
> > strange. You've essentially got two independent ZooKeeper clusters set
> up.
> > They won't communicate with each other at all. I'd presume that if you've
> > got a database shared between your ManifoldCF instances, that you'd also
> > want your ZooKeeper cluster to be shared between these two instances.
> >
> > In this case, you'd normally run your ZooKeeper nodes across both sites
> (2
> > + 1, or 3 + 2) and then have both of your client (ManifoldCF) nodes
> > communicate with this cluster.
> >
> > Having said that, there may be a reason that ManifoldCF uses two
> > independent clusters that don't communicate with each other, I'm not
> sure.
> >
> > cheers
> > Cam
> >
> >
> >
> > On Fri, Jul 4, 2014 at 7:17 AM, lalit jangra <la...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I am a newbie in Zookeeper are so please apologize for any not so
> > relevant
> > > question asked.
> > >
> > > I am running clustered Apache ManifoldCF on two nodes which are
> pointing
> > to
> > > same database. But i need synchronization between both nodes so avoid
> > race
> > > condition while both nodes are working together.
> > >
> > > For same, as per Apache ManifoldCF guidelines, i am using Zookeeper to
> > > manage synchronization in cluster.Also as per ManifoldCF & Zookeeper
> > > instructions, i have setup & configured two set of three Zookeeper
> server
> > > at both nodes which are working fine.
> > >
> > > But i have a clustered setup where both ManifoldCF nodes point to same
> > DB ,
> > > i am still clueless about how should i proceed to connect Zookeeper
> nodes
> > > at both servers to make a single setup.
> > >
> > > Also i am not sure whether it is necessary or not but then the question
> > > arises that how two servers & Zookeeper nodes in cluster will talk to
> > each
> > > other?
> > >
> > > So normally what should be the steps taken once zookeeper servers are
> > setup
> > > to maintain synchronization.
> > >
> > > Regards,
> > > Lalit Jangra.
> > >
> >
>
>
>
> --
> Regards,
> Lalit Jangra.
>

Re: Running Zookeeper in clustered setup/

[lalit jangra <la...@gmail.com>]

Thanks Cam,

So i should have either 2+1 or 3+2 Zookeeper servers on these tow nodes to
which both nodes are talking?

In this case, what should i put in zoo.cfg file for host values as
currently i am using localhost at both nodes.

server.1=localhost:2888:3888
server.1=localhost:2889:3889
server.1=localhost:28908:3890

Again how these 3 nodes will form a cluster?

Regards.


On Thu, Jul 3, 2014 at 10:27 PM, Cameron McKenzie <mc...@gmail.com>
wrote:

> hey Lalit,
> I'm not familiar with Apache ManifoldCF, but the configuration sounds a bit
> strange. You've essentially got two independent ZooKeeper clusters set up.
> They won't communicate with each other at all. I'd presume that if you've
> got a database shared between your ManifoldCF instances, that you'd also
> want your ZooKeeper cluster to be shared between these two instances.
>
> In this case, you'd normally run your ZooKeeper nodes across both sites (2
> + 1, or 3 + 2) and then have both of your client (ManifoldCF) nodes
> communicate with this cluster.
>
> Having said that, there may be a reason that ManifoldCF uses two
> independent clusters that don't communicate with each other, I'm not sure.
>
> cheers
> Cam
>
>
>
> On Fri, Jul 4, 2014 at 7:17 AM, lalit jangra <la...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am a newbie in Zookeeper are so please apologize for any not so
> relevant
> > question asked.
> >
> > I am running clustered Apache ManifoldCF on two nodes which are pointing
> to
> > same database. But i need synchronization between both nodes so avoid
> race
> > condition while both nodes are working together.
> >
> > For same, as per Apache ManifoldCF guidelines, i am using Zookeeper to
> > manage synchronization in cluster.Also as per ManifoldCF & Zookeeper
> > instructions, i have setup & configured two set of three Zookeeper server
> > at both nodes which are working fine.
> >
> > But i have a clustered setup where both ManifoldCF nodes point to same
> DB ,
> > i am still clueless about how should i proceed to connect Zookeeper nodes
> > at both servers to make a single setup.
> >
> > Also i am not sure whether it is necessary or not but then the question
> > arises that how two servers & Zookeeper nodes in cluster will talk to
> each
> > other?
> >
> > So normally what should be the steps taken once zookeeper servers are
> setup
> > to maintain synchronization.
> >
> > Regards,
> > Lalit Jangra.
> >
>



-- 
Regards,
Lalit Jangra.

Re: Running Zookeeper in clustered setup/

[Cameron McKenzie <mc...@gmail.com>]

hey Lalit,
I'm not familiar with Apache ManifoldCF, but the configuration sounds a bit
strange. You've essentially got two independent ZooKeeper clusters set up.
They won't communicate with each other at all. I'd presume that if you've
got a database shared between your ManifoldCF instances, that you'd also
want your ZooKeeper cluster to be shared between these two instances.

In this case, you'd normally run your ZooKeeper nodes across both sites (2
+ 1, or 3 + 2) and then have both of your client (ManifoldCF) nodes
communicate with this cluster.

Having said that, there may be a reason that ManifoldCF uses two
independent clusters that don't communicate with each other, I'm not sure.

cheers
Cam



On Fri, Jul 4, 2014 at 7:17 AM, lalit jangra <la...@gmail.com>
wrote:

> Hi,
>
> I am a newbie in Zookeeper are so please apologize for any not so relevant
> question asked.
>
> I am running clustered Apache ManifoldCF on two nodes which are pointing to
> same database. But i need synchronization between both nodes so avoid race
> condition while both nodes are working together.
>
> For same, as per Apache ManifoldCF guidelines, i am using Zookeeper to
> manage synchronization in cluster.Also as per ManifoldCF & Zookeeper
> instructions, i have setup & configured two set of three Zookeeper server
> at both nodes which are working fine.
>
> But i have a clustered setup where both ManifoldCF nodes point to same DB ,
> i am still clueless about how should i proceed to connect Zookeeper nodes
> at both servers to make a single setup.
>
> Also i am not sure whether it is necessary or not but then the question
> arises that how two servers & Zookeeper nodes in cluster will talk to each
> other?
>
> So normally what should be the steps taken once zookeeper servers are setup
> to maintain synchronization.
>
> Regards,
> Lalit Jangra.
>