federated cluster

[Visioner Sadak <vi...@gmail.com>]

Hello friends i have configured a 3 node federated hadoop cluster with 2
namenodes and 3 datanodes

2 namenodes -- ip1,ip2

3 datanodes   --ip1,ip2,ip3

wht nodes shud i make as my secondary namenode,is it mandatory to have
resource manager and node manager,i m using hadoop just for storing purpose
so i dunt need them

some data goes into namenode 1 which is ip1, some goes in to namenode 2
which is ip2

i hope i cant access the data of nn1 using nn2 ip or any nn ip in my
cluster right,
i can access data only using the nn1 ip for nn1 data ....correct me if i am
wrong...

Re: federated cluster

[Visioner Sadak <vi...@gmail.com>]

Hey Thanks a ton Harsh You Rock !!!!!!!!!!

On Mon, Nov 5, 2012 at 7:26 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Visioner,
>
> If you aren't using HA, you will need two SNNs - one that checkpoints
> IP1 and the other for IP2. If you lack more hardware, you may probably
> reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
> wouldn't work so well after a while (when metadata at each NN has
> grown much), but for starters this is good enough I guess.
>
> The ResourceManager/NodeManager are YARN daemons useful for processing
> data. If your need is just storage, you are correct in not needing
> them deployed.
>
> You are also correct on the federated NNs part. If a file resides in
> the NS of IP1, you need to access IP1 to fetch its details.
>
> On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com>
> wrote:
> > Hello friends i have configured a 3 node federated hadoop cluster with 2
> > namenodes and 3 datanodes
> >
> > 2 namenodes -- ip1,ip2
> >
> > 3 datanodes   --ip1,ip2,ip3
> >
> > wht nodes shud i make as my secondary namenode,is it mandatory to have
> > resource manager and node manager,i m using hadoop just for storing
> purpose
> > so i dunt need them
> >
> > some data goes into namenode 1 which is ip1, some goes in to namenode 2
> > which is ip2
> >
> > i hope i cant access the data of nn1 using nn2 ip or any nn ip in my
> cluster
> > right,
> > i can access data only using the nn1 ip for nn1 data ....correct me if i
> am
> > wrong...
> >
> >
> >
>
>
>
> --
> Harsh J
>

Re: federated cluster

[Visioner Sadak <vi...@gmail.com>]

Hey Thanks a ton Harsh You Rock !!!!!!!!!!

On Mon, Nov 5, 2012 at 7:26 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Visioner,
>
> If you aren't using HA, you will need two SNNs - one that checkpoints
> IP1 and the other for IP2. If you lack more hardware, you may probably
> reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
> wouldn't work so well after a while (when metadata at each NN has
> grown much), but for starters this is good enough I guess.
>
> The ResourceManager/NodeManager are YARN daemons useful for processing
> data. If your need is just storage, you are correct in not needing
> them deployed.
>
> You are also correct on the federated NNs part. If a file resides in
> the NS of IP1, you need to access IP1 to fetch its details.
>
> On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com>
> wrote:
> > Hello friends i have configured a 3 node federated hadoop cluster with 2
> > namenodes and 3 datanodes
> >
> > 2 namenodes -- ip1,ip2
> >
> > 3 datanodes   --ip1,ip2,ip3
> >
> > wht nodes shud i make as my secondary namenode,is it mandatory to have
> > resource manager and node manager,i m using hadoop just for storing
> purpose
> > so i dunt need them
> >
> > some data goes into namenode 1 which is ip1, some goes in to namenode 2
> > which is ip2
> >
> > i hope i cant access the data of nn1 using nn2 ip or any nn ip in my
> cluster
> > right,
> > i can access data only using the nn1 ip for nn1 data ....correct me if i
> am
> > wrong...
> >
> >
> >
>
>
>
> --
> Harsh J
>

Re: federated cluster

[Visioner Sadak <vi...@gmail.com>]

Hey Thanks a ton Harsh You Rock !!!!!!!!!!

On Mon, Nov 5, 2012 at 7:26 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Visioner,
>
> If you aren't using HA, you will need two SNNs - one that checkpoints
> IP1 and the other for IP2. If you lack more hardware, you may probably
> reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
> wouldn't work so well after a while (when metadata at each NN has
> grown much), but for starters this is good enough I guess.
>
> The ResourceManager/NodeManager are YARN daemons useful for processing
> data. If your need is just storage, you are correct in not needing
> them deployed.
>
> You are also correct on the federated NNs part. If a file resides in
> the NS of IP1, you need to access IP1 to fetch its details.
>
> On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com>
> wrote:
> > Hello friends i have configured a 3 node federated hadoop cluster with 2
> > namenodes and 3 datanodes
> >
> > 2 namenodes -- ip1,ip2
> >
> > 3 datanodes   --ip1,ip2,ip3
> >
> > wht nodes shud i make as my secondary namenode,is it mandatory to have
> > resource manager and node manager,i m using hadoop just for storing
> purpose
> > so i dunt need them
> >
> > some data goes into namenode 1 which is ip1, some goes in to namenode 2
> > which is ip2
> >
> > i hope i cant access the data of nn1 using nn2 ip or any nn ip in my
> cluster
> > right,
> > i can access data only using the nn1 ip for nn1 data ....correct me if i
> am
> > wrong...
> >
> >
> >
>
>
>
> --
> Harsh J
>

Re: federated cluster

[Visioner Sadak <vi...@gmail.com>]

Hey Thanks a ton Harsh You Rock !!!!!!!!!!

On Mon, Nov 5, 2012 at 7:26 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Visioner,
>
> If you aren't using HA, you will need two SNNs - one that checkpoints
> IP1 and the other for IP2. If you lack more hardware, you may probably
> reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
> wouldn't work so well after a while (when metadata at each NN has
> grown much), but for starters this is good enough I guess.
>
> The ResourceManager/NodeManager are YARN daemons useful for processing
> data. If your need is just storage, you are correct in not needing
> them deployed.
>
> You are also correct on the federated NNs part. If a file resides in
> the NS of IP1, you need to access IP1 to fetch its details.
>
> On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com>
> wrote:
> > Hello friends i have configured a 3 node federated hadoop cluster with 2
> > namenodes and 3 datanodes
> >
> > 2 namenodes -- ip1,ip2
> >
> > 3 datanodes   --ip1,ip2,ip3
> >
> > wht nodes shud i make as my secondary namenode,is it mandatory to have
> > resource manager and node manager,i m using hadoop just for storing
> purpose
> > so i dunt need them
> >
> > some data goes into namenode 1 which is ip1, some goes in to namenode 2
> > which is ip2
> >
> > i hope i cant access the data of nn1 using nn2 ip or any nn ip in my
> cluster
> > right,
> > i can access data only using the nn1 ip for nn1 data ....correct me if i
> am
> > wrong...
> >
> >
> >
>
>
>
> --
> Harsh J
>

Re: federated cluster

[Harsh J <ha...@cloudera.com>]

Hi Visioner,

If you aren't using HA, you will need two SNNs - one that checkpoints
IP1 and the other for IP2. If you lack more hardware, you may probably
reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
wouldn't work so well after a while (when metadata at each NN has
grown much), but for starters this is good enough I guess.

The ResourceManager/NodeManager are YARN daemons useful for processing
data. If your need is just storage, you are correct in not needing
them deployed.

You are also correct on the federated NNs part. If a file resides in
the NS of IP1, you need to access IP1 to fetch its details.

On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com> wrote:
> Hello friends i have configured a 3 node federated hadoop cluster with 2
> namenodes and 3 datanodes
>
> 2 namenodes -- ip1,ip2
>
> 3 datanodes   --ip1,ip2,ip3
>
> wht nodes shud i make as my secondary namenode,is it mandatory to have
> resource manager and node manager,i m using hadoop just for storing purpose
> so i dunt need them
>
> some data goes into namenode 1 which is ip1, some goes in to namenode 2
> which is ip2
>
> i hope i cant access the data of nn1 using nn2 ip or any nn ip in my cluster
> right,
> i can access data only using the nn1 ip for nn1 data ....correct me if i am
> wrong...
>
>
>



-- 
Harsh J

Re: federated cluster

[Harsh J <ha...@cloudera.com>]

Hi Visioner,

If you aren't using HA, you will need two SNNs - one that checkpoints
IP1 and the other for IP2. If you lack more hardware, you may probably
reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
wouldn't work so well after a while (when metadata at each NN has
grown much), but for starters this is good enough I guess.

The ResourceManager/NodeManager are YARN daemons useful for processing
data. If your need is just storage, you are correct in not needing
them deployed.

You are also correct on the federated NNs part. If a file resides in
the NS of IP1, you need to access IP1 to fetch its details.

On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com> wrote:
> Hello friends i have configured a 3 node federated hadoop cluster with 2
> namenodes and 3 datanodes
>
> 2 namenodes -- ip1,ip2
>
> 3 datanodes   --ip1,ip2,ip3
>
> wht nodes shud i make as my secondary namenode,is it mandatory to have
> resource manager and node manager,i m using hadoop just for storing purpose
> so i dunt need them
>
> some data goes into namenode 1 which is ip1, some goes in to namenode 2
> which is ip2
>
> i hope i cant access the data of nn1 using nn2 ip or any nn ip in my cluster
> right,
> i can access data only using the nn1 ip for nn1 data ....correct me if i am
> wrong...
>
>
>



-- 
Harsh J

Re: federated cluster

[Harsh J <ha...@cloudera.com>]

Hi Visioner,

If you aren't using HA, you will need two SNNs - one that checkpoints
IP1 and the other for IP2. If you lack more hardware, you may probably
reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
wouldn't work so well after a while (when metadata at each NN has
grown much), but for starters this is good enough I guess.

The ResourceManager/NodeManager are YARN daemons useful for processing
data. If your need is just storage, you are correct in not needing
them deployed.

You are also correct on the federated NNs part. If a file resides in
the NS of IP1, you need to access IP1 to fetch its details.

On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com> wrote:
> Hello friends i have configured a 3 node federated hadoop cluster with 2
> namenodes and 3 datanodes
>
> 2 namenodes -- ip1,ip2
>
> 3 datanodes   --ip1,ip2,ip3
>
> wht nodes shud i make as my secondary namenode,is it mandatory to have
> resource manager and node manager,i m using hadoop just for storing purpose
> so i dunt need them
>
> some data goes into namenode 1 which is ip1, some goes in to namenode 2
> which is ip2
>
> i hope i cant access the data of nn1 using nn2 ip or any nn ip in my cluster
> right,
> i can access data only using the nn1 ip for nn1 data ....correct me if i am
> wrong...
>
>
>



-- 
Harsh J

Re: federated cluster

[Harsh J <ha...@cloudera.com>]

Hi Visioner,

If you aren't using HA, you will need two SNNs - one that checkpoints
IP1 and the other for IP2. If you lack more hardware, you may probably
reside them on each other (i.e. SNN of IP2 on IP1, etc.) although this
wouldn't work so well after a while (when metadata at each NN has
grown much), but for starters this is good enough I guess.

The ResourceManager/NodeManager are YARN daemons useful for processing
data. If your need is just storage, you are correct in not needing
them deployed.

You are also correct on the federated NNs part. If a file resides in
the NS of IP1, you need to access IP1 to fetch its details.

On Mon, Nov 5, 2012 at 5:50 PM, Visioner Sadak <vi...@gmail.com> wrote:
> Hello friends i have configured a 3 node federated hadoop cluster with 2
> namenodes and 3 datanodes
>
> 2 namenodes -- ip1,ip2
>
> 3 datanodes   --ip1,ip2,ip3
>
> wht nodes shud i make as my secondary namenode,is it mandatory to have
> resource manager and node manager,i m using hadoop just for storing purpose
> so i dunt need them
>
> some data goes into namenode 1 which is ip1, some goes in to namenode 2
> which is ip2
>
> i hope i cant access the data of nn1 using nn2 ip or any nn ip in my cluster
> right,
> i can access data only using the nn1 ip for nn1 data ....correct me if i am
> wrong...
>
>
>



-- 
Harsh J