Dynamically specify slave nodes

[Francesco De Luca <f....@gmail.com>]

Is it possible to dynamically specify the slave nodes  instead to specify
them statically through the configuration files?

I want to build a dynamic environment in which each node can enter and exit,
so i need to specify the new
entered node as a new slave node for hadoop framework

Thanks

Re: Dynamically specify slave nodes

[Francesco De Luca <f....@gmail.com>]

Dear Harsh,

my purpose is to build a java map reduce framework based on a p2p model
to provide dynamic failure recovery of master, slave and task.
I've developed the p2p logic capable to track all the information needed to
allow these recovery,
but now I need to add a map reduce module to execute a real mapreduce job.
I'm investigating if i can adopt hadoop mapreduce and track all the
information my system needs,
or if i need to re implement a simple mapreduce framework from scratch.

I hope I was clear enough :-) !
Thanks a lot.

2011/5/24 Harsh J <ha...@cloudera.com>

> Francesco,
>
> The system is already dynamic right now, unless a configuration limits
> that. Slave nodes may enter/exit any time. But you have to keep in
> mind that the loss of a DataNode can lead to a lot of network traffic
> for rebalancing/re-replicating blocks lost. A TaskTracker loss is not
> so expensive to the cluster, OTOH.
>
> Merely starting up a DN/TT individually on a machine on the network
> with the right master hosts configuration will see it automatically
> enter the NN/JT cluster.
>
> Is this what you are looking to do?
>
> On Tue, May 24, 2011 at 4:13 PM, Francesco De Luca <f....@gmail.com>
> wrote:
> > Is it possible to dynamically specify the slave nodes  instead to specify
> > them statically through the configuration files?
> >
> > I want to build a dynamic environment in which each node can enter and
> exit,
> > so i need to specify the new
> > entered node as a new slave node for hadoop framework
> >
> > Thanks
> >
>
>
>
> --
> Harsh J
>

Re: Dynamically specify slave nodes

[Harsh J <ha...@cloudera.com>]

Francesco,

The system is already dynamic right now, unless a configuration limits
that. Slave nodes may enter/exit any time. But you have to keep in
mind that the loss of a DataNode can lead to a lot of network traffic
for rebalancing/re-replicating blocks lost. A TaskTracker loss is not
so expensive to the cluster, OTOH.

Merely starting up a DN/TT individually on a machine on the network
with the right master hosts configuration will see it automatically
enter the NN/JT cluster.

Is this what you are looking to do?

On Tue, May 24, 2011 at 4:13 PM, Francesco De Luca <f....@gmail.com> wrote:
> Is it possible to dynamically specify the slave nodes  instead to specify
> them statically through the configuration files?
>
> I want to build a dynamic environment in which each node can enter and exit,
> so i need to specify the new
> entered node as a new slave node for hadoop framework
>
> Thanks
>



-- 
Harsh J