You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by nitesh bhatia <ni...@gmail.com> on 2009/01/25 17:45:14 UTC
Zeroconf for hadoop
Hi
Apple provides opensource discovery service called Bonjour (zeroconf). Is it
possible to integrate Zeroconf with Hadoop so that discovery of nodes become
automatic ? Presently for setting up multi-node cluster we need to add IPs
manually. Integrating it with bonjour can make this process automatic.
--nitesh
--
Nitesh Bhatia
Dhirubhai Ambani Institute of Information & Communication Technology
Gandhinagar
Gujarat
"Life is never perfect. It just depends where you draw the line."
visit:
http://www.awaaaz.com - connecting through music
http://www.volstreet.com - lets volunteer for better tomorrow
http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
Re: Zeroconf for hadoop
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
nitesh bhatia wrote:
> Hi
> Apple provides opensource discovery service called Bonjour (zeroconf). Is it
> possible to integrate Zeroconf with Hadoop so that discovery of nodes become
> automatic ? Presently for setting up multi-node cluster we need to add IPs
> manually. Integrating it with bonjour can make this process automatic.
It would be nice to have. Note that it is the slaves (DataNodes,
TaskTrackers) that need to do the discovery. NameNode just passively
accepts the DataNodes that want to be part of the cluster.
In that sense, NN should announce itself and DNs try to find where the
NN is. It will nice to have zeroconf feature in some form and we might
discover many more uses for it. Of course, a cluster should not require it.
Raghu.
Re: Zeroconf for hadoop
Posted by Steve Loughran <st...@apache.org>.
Doug Cutting wrote:
> Owen O'Malley wrote:
>> allssh -h node1000-3000 bin/hadoop-daemon.sh start tasktracker
>>
>> and it will use ssh in parallel to connect to every node between
>> node1000 and node3000. Our's is a mess, but it would be great if
>> someone contributed a script like that. *smile*
>
> It would be a one-line change to bin/slaves.sh to have it filter hosts
> by a regex.
>
> Note that bin/slaves.sh can have problems with larger clusters (>~100
> nodes) since a single shell has trouble handling the i/o from 100
> sub-processes, and ssh connections will start timing out. That's the
> point of the HADOOP_SLAVE_SLEEP parameter, to meter the rate that
> sub-processes are spawned. A better solution might be too sleep if the
> number of sub-processes exceeds some limit, e.g.:
>
> while [[ `jobs | wc -l` > 10 ]]; do sleep 1 ; done
>
> Doug
The trick there is for your script to pick the first couple of nodes and
give them half the work, they do the same thing down the tree and you
end up with the cluster booting itself at some rate that includes
log2(N) somewhere in the equations.
Re: Zeroconf for hadoop
Posted by Doug Cutting <cu...@apache.org>.
Owen O'Malley wrote:
> allssh -h node1000-3000 bin/hadoop-daemon.sh start tasktracker
>
> and it will use ssh in parallel to connect to every node between
> node1000 and node3000. Our's is a mess, but it would be great if someone
> contributed a script like that. *smile*
It would be a one-line change to bin/slaves.sh to have it filter hosts
by a regex.
Note that bin/slaves.sh can have problems with larger clusters (>~100
nodes) since a single shell has trouble handling the i/o from 100
sub-processes, and ssh connections will start timing out. That's the
point of the HADOOP_SLAVE_SLEEP parameter, to meter the rate that
sub-processes are spawned. A better solution might be too sleep if the
number of sub-processes exceeds some limit, e.g.:
while [[ `jobs | wc -l` > 10 ]]; do sleep 1 ; done
Doug
Re: Zeroconf for hadoop
Posted by Ted Dunning <te...@gmail.com>.
A big positive vote for Zookeeper.
The most salient aspect of my experience using zookeeper is that
coordination and heartbeats and discovery and failure notifications all
become nearly trivial. The most amazing thing about ZK isn't the code that
you write, it is all the code that you never have write.
On Wed, Jan 28, 2009 at 11:01 AM, Patrick Hunt <ph...@apache.org> wrote:
> Owen O'Malley wrote:
>
>> On Jan 25, 2009, at 2:02 PM, nitesh bhatia wrote:
>>
>> Apple provides opensource discovery service called Bonjour (zeroconf).
>>>
>>
>> I don't know enough about Zeroconf to be able to answer definitively, but
>> I suspect the hardest bit would be figuring out the approach. Of course
>> Hadoop has to continue to work on other platforms, so cross-platform
>> strategies are better.
>>
>
> Take a look at ZooKeeper (a sub-project of Hadoop):
> http://hadoop.apache.org/zookeeper/
>
> Among other features ZooKeeper provides group membership and dynamic
> configuration support -- you could modify the various Hadoop processes to
> query & register with ZooKeeper when they come up. This could be used for
> node/service discovery as well as auto configuration of the processes.
>
> Patrick
>
--
Ted Dunning, CTO
DeepDyve
4600 Bohannon Drive, Suite 220
Menlo Park, CA 94025
www.deepdyve.com
650-324-0110, ext. 738
858-414-0013 (m)
Re: Zeroconf for hadoop
Posted by Patrick Hunt <ph...@apache.org>.
Owen O'Malley wrote:
> On Jan 25, 2009, at 2:02 PM, nitesh bhatia wrote:
>
>> Apple provides opensource discovery service called Bonjour (zeroconf).
>
> I don't know enough about Zeroconf to be able to answer definitively,
> but I suspect the hardest bit would be figuring out the approach. Of
> course Hadoop has to continue to work on other platforms, so
> cross-platform strategies are better.
Take a look at ZooKeeper (a sub-project of Hadoop):
http://hadoop.apache.org/zookeeper/
Among other features ZooKeeper provides group membership and dynamic
configuration support -- you could modify the various Hadoop processes
to query & register with ZooKeeper when they come up. This could be used
for node/service discovery as well as auto configuration of the processes.
Patrick
Re: Zeroconf for hadoop
Posted by Owen O'Malley <om...@apache.org>.
On Jan 25, 2009, at 2:02 PM, nitesh bhatia wrote:
> Apple provides opensource discovery service called Bonjour (zeroconf).
I don't know enough about Zeroconf to be able to answer definitively,
but I suspect the hardest bit would be figuring out the approach. Of
course Hadoop has to continue to work on other platforms, so cross-
platform strategies are better.
> Presently for setting up multi-node cluster we need to add IPs
> manually.
To the slaves list? The slaves list is only used to ssh to the hosts.
At Yahoo, we use a parallel ssh perl script that takes ranges of
hosts, so you issue commands like:
allssh -h node1000-3000 bin/hadoop-daemon.sh start tasktracker
and it will use ssh in parallel to connect to every node between
node1000 and node3000. Our's is a mess, but it would be great if
someone contributed a script like that. *smile*
-- Owen
Zeroconf for hadoop
Posted by nitesh bhatia <ni...@gmail.com>.
Hi
Apple provides opensource discovery service called Bonjour (zeroconf). Is it
possible to integrate Zeroconf with Hadoop so that discovery of nodes become
automatic ? Presently for setting up multi-node cluster we need to add IPs
manually. Integrating it with bonjour can make this process automatic.
--nitesh
--
Nitesh Bhatia
Dhirubhai Ambani Institute of Information & Communication Technology
Gandhinagar
Gujarat
"Life is never perfect. It just depends where you draw the line."
visit:
http://www.awaaaz.com - connecting through music
http://www.volstreet.com - lets volunteer for better tomorrow
http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
Re: Zeroconf for hadoop
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Nitay wrote:
> Why not use the distributed coordination service ZooKeeper? When nodes come
> up they write some ephemeral file in a known ZooKeeper directory and anyone
> who's interested, i.e. NameNode, can put a watch on the directory and get
> notified when new children come up.
NameNode does not do active discovery. It is the DataNodes that contact
NameNode about their presence. So with ZooKeeper or zeroconf, DataNode
should be able to discover who their NN is and connect to it.
Raghu.
> On Mon, Jan 26, 2009 at 10:59 AM, Allen Wittenauer <aw...@yahoo-inc.com> wrote:
>
>>
>>
>> On 1/25/09 8:45 AM, "nitesh bhatia" <ni...@gmail.com> wrote:
>>> Apple provides opensource discovery service called Bonjour (zeroconf). Is
>> it
>>> possible to integrate Zeroconf with Hadoop so that discovery of nodes
>> become
>>> automatic ? Presently for setting up multi-node cluster we need to add
>> IPs
>>> manually. Integrating it with bonjour can make this process automatic.
>> How do you deal with multiple grids?
>>
>> How do you deal with security?
>>
>>
>
Re: Zeroconf for hadoop
Posted by Vadim Zaliva <kr...@gmail.com>.
On Mon, Jan 26, 2009 at 11:22, Edward Capriolo <ed...@gmail.com> wrote:
> Zeroconf is more focused on simplicity then security. One of the
> original problems that may have been fixes is that any program can
> announce any service. IE my laptop can announce that it is the DNS for
> google.com etc.
I see two distinct tasks here:
1. Discovery
2. Authorization
Zeroconf would allow easy discovery of potential new nodes. Then if
they are configured with proper security credentials they could be
used in a cluster.
Vadim
Re: Zeroconf for hadoop
Posted by Steve Loughran <st...@apache.org>.
Edward Capriolo wrote:
> Zeroconf is more focused on simplicity then security. One of the
> original problems that may have been fixes is that any program can
> announce any service. IE my laptop can announce that it is the DNS for
> google.com etc.
>
-1 to zeroconf as it is way too chatty. Every DNS lookup is mcast, in a
busy network a lot of CPU time is spent discarding requests. Nor does it
handle failure that well. It's OK on a home LAN to find a music player,
but not what you want for a HA infrastructure in the datacentre,
Our LAN discovery tool -Anubis -uses mcast only to do the initial
discovery, then they have voting and things to select a nominated server
that everyone just unicasts too at that point; failure of that
node/network partition triggers a rebinding.
See: http://wiki.smartfrog.org/wiki/display/sf/Anubis ; the paper
discusses some of the fun you have, though that paper doesn't also
include clock drift issue you can encounter when running Xen or
VMWare-hosted nodes.
> I want to mention a related topic to the list. People are approaching
> the auto-discovery in a number of ways jira. There are a few ways I
> can think of to discover hadoop. A very simple way might be to publish
> the configuration over a web interface. I use a network storage system
> called gluster-fs. Gluster can be configured so the server holds the
> configuration for each client. If the hadoop name node held the entire
> configuration for all the nodes the namenode would only need to be
> aware of the namenode and it could retrieve its configuration from it.
>
> Having a central configuration management or a discovery system would
> be very useful. HOD is what I think to be the closest thing it is more
> of a top down deployment system.
Allen is a fan of a well managed cluster; he pushes out Hadoop as RPMs
via PXE and Kickstart and uses LDAP as the central CM tool. I am
currently exploring bringing up virtual clusters by
* putting the relevant RPMs out to all nodes; same files/conf for
every node,
* having custom configs for Namenode and job tracker; everything else
becomes a Datanode with a task tracker bound to the masters.
I will start worrying about discovery afterwards, because without the
ability for the Job Tracker or Namenode to do failover to a fallback Job
Tracker or Namenode, you don't really need so much in the way of dynamic
cluster binding.
-steve
Re: Zeroconf for hadoop
Posted by Edward Capriolo <ed...@gmail.com>.
Zeroconf is more focused on simplicity then security. One of the
original problems that may have been fixes is that any program can
announce any service. IE my laptop can announce that it is the DNS for
google.com etc.
I want to mention a related topic to the list. People are approaching
the auto-discovery in a number of ways jira. There are a few ways I
can think of to discover hadoop. A very simple way might be to publish
the configuration over a web interface. I use a network storage system
called gluster-fs. Gluster can be configured so the server holds the
configuration for each client. If the hadoop name node held the entire
configuration for all the nodes the namenode would only need to be
aware of the namenode and it could retrieve its configuration from it.
Having a central configuration management or a discovery system would
be very useful. HOD is what I think to be the closest thing it is more
of a top down deployment system.
Re: Zeroconf for hadoop
Posted by Nitay <ni...@gmail.com>.
Why not use the distributed coordination service ZooKeeper? When nodes come
up they write some ephemeral file in a known ZooKeeper directory and anyone
who's interested, i.e. NameNode, can put a watch on the directory and get
notified when new children come up.
On Mon, Jan 26, 2009 at 10:59 AM, Allen Wittenauer <aw...@yahoo-inc.com> wrote:
>
>
>
> On 1/25/09 8:45 AM, "nitesh bhatia" <ni...@gmail.com> wrote:
> > Apple provides opensource discovery service called Bonjour (zeroconf). Is
> it
> > possible to integrate Zeroconf with Hadoop so that discovery of nodes
> become
> > automatic ? Presently for setting up multi-node cluster we need to add
> IPs
> > manually. Integrating it with bonjour can make this process automatic.
>
> How do you deal with multiple grids?
>
> How do you deal with security?
>
>
Re: Zeroconf for hadoop
Posted by nitesh bhatia <ni...@gmail.com>.
For a closed uniform system (yahoo, google), this can work best. This can
provide plug-n-play type of system. Through this we can change clusters to
dynamic grids. But I am not sure of outcome so far, I am reading the
documentation.
--nitesh
On Mon, Jan 26, 2009 at 1:59 PM, Allen Wittenauer <aw...@yahoo-inc.com> wrote:
>
>
>
> On 1/25/09 8:45 AM, "nitesh bhatia" <ni...@gmail.com> wrote:
> > Apple provides opensource discovery service called Bonjour (zeroconf). Is
> it
> > possible to integrate Zeroconf with Hadoop so that discovery of nodes
> become
> > automatic ? Presently for setting up multi-node cluster we need to add
> IPs
> > manually. Integrating it with bonjour can make this process automatic.
>
> How do you deal with multiple grids?
>
> How do you deal with security?
>
>
--
Nitesh Bhatia
Dhirubhai Ambani Institute of Information & Communication Technology
Gandhinagar
Gujarat
"Life is never perfect. It just depends where you draw the line."
visit:
http://www.awaaaz.com - connecting through music
http://www.volstreet.com - lets volunteer for better tomorrow
http://www.instibuzz.com - Voice opinions, Transact easily, Have fun
Re: Zeroconf for hadoop
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 1/25/09 8:45 AM, "nitesh bhatia" <ni...@gmail.com> wrote:
> Apple provides opensource discovery service called Bonjour (zeroconf). Is it
> possible to integrate Zeroconf with Hadoop so that discovery of nodes become
> automatic ? Presently for setting up multi-node cluster we need to add IPs
> manually. Integrating it with bonjour can make this process automatic.
How do you deal with multiple grids?
How do you deal with security?