You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by David Parks <da...@yahoo.com> on 2013/03/18 11:24:20 UTC
On a small cluster can we double up namenode/master with tasktrackers?
I want 20 servers, I got 7, so I want to make the most of the 7 I have. Each
of the 7 servers have: 24GB of ram, 4TB, and 8 cores.
Would it be terribly unwise of me to Run such a configuration:
. Server #1: NameNode + Master + TaskTracker(reduced
slots)
. Server #2: CheckpointNode(aka Secondary Name Node) +
TaskTracker(slightly reduced slots)
. Server #3: TaskTracker
. Server #4: TaskTracker
. Server #5: TaskTracker
. Server #6: TaskTracker
. Server #7: TaskTracker
Did I miss anything? Did I shoot myself in the foot anywhere?
RE: On a small cluster can we double up namenode/master with tasktrackers?
Posted by David Parks <da...@yahoo.com>.
Good points all,
The mapreduce jobs are, well. intensive. We've got a whole variety, but
typically I see them use a lot of CPU, a lot of Disk, and upon occasion a
whole bunch of Network bandwidth. Duh right? J
The master node is mostly CPU intensive right? We're using LXC to segregate
(psudo-virtualize) our environments for ease of development and management.
I'm looking into whether I can use LXC's quota system to guarantee a certain
level of CPU resources to the container where the master node is housed. If
I can do that I guess we wouldn't have any issue here.
Thanks!
David
From: Jens Scheidtmann [mailto:jens.scheidtmann@gmail.com]
Sent: Tuesday, March 19, 2013 3:12 PM
To: user@hadoop.apache.org
Subject: Re: On a small cluster can we double up namenode/master with
tasktrackers?
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
RE: On a small cluster can we double up namenode/master with tasktrackers?
Posted by David Parks <da...@yahoo.com>.
Good points all,
The mapreduce jobs are, well. intensive. We've got a whole variety, but
typically I see them use a lot of CPU, a lot of Disk, and upon occasion a
whole bunch of Network bandwidth. Duh right? J
The master node is mostly CPU intensive right? We're using LXC to segregate
(psudo-virtualize) our environments for ease of development and management.
I'm looking into whether I can use LXC's quota system to guarantee a certain
level of CPU resources to the container where the master node is housed. If
I can do that I guess we wouldn't have any issue here.
Thanks!
David
From: Jens Scheidtmann [mailto:jens.scheidtmann@gmail.com]
Sent: Tuesday, March 19, 2013 3:12 PM
To: user@hadoop.apache.org
Subject: Re: On a small cluster can we double up namenode/master with
tasktrackers?
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
RE: On a small cluster can we double up namenode/master with tasktrackers?
Posted by David Parks <da...@yahoo.com>.
Good points all,
The mapreduce jobs are, well. intensive. We've got a whole variety, but
typically I see them use a lot of CPU, a lot of Disk, and upon occasion a
whole bunch of Network bandwidth. Duh right? J
The master node is mostly CPU intensive right? We're using LXC to segregate
(psudo-virtualize) our environments for ease of development and management.
I'm looking into whether I can use LXC's quota system to guarantee a certain
level of CPU resources to the container where the master node is housed. If
I can do that I guess we wouldn't have any issue here.
Thanks!
David
From: Jens Scheidtmann [mailto:jens.scheidtmann@gmail.com]
Sent: Tuesday, March 19, 2013 3:12 PM
To: user@hadoop.apache.org
Subject: Re: On a small cluster can we double up namenode/master with
tasktrackers?
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
RE: On a small cluster can we double up namenode/master with tasktrackers?
Posted by David Parks <da...@yahoo.com>.
Good points all,
The mapreduce jobs are, well. intensive. We've got a whole variety, but
typically I see them use a lot of CPU, a lot of Disk, and upon occasion a
whole bunch of Network bandwidth. Duh right? J
The master node is mostly CPU intensive right? We're using LXC to segregate
(psudo-virtualize) our environments for ease of development and management.
I'm looking into whether I can use LXC's quota system to guarantee a certain
level of CPU resources to the container where the master node is housed. If
I can do that I guess we wouldn't have any issue here.
Thanks!
David
From: Jens Scheidtmann [mailto:jens.scheidtmann@gmail.com]
Sent: Tuesday, March 19, 2013 3:12 PM
To: user@hadoop.apache.org
Subject: Re: On a small cluster can we double up namenode/master with
tasktrackers?
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Jens Scheidtmann <je...@gmail.com>.
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Harsh J <ha...@cloudera.com>.
I think its fine to keep it so long as you also monitor the nodes for
issues during heavy load times and tweak slave configs appropriately
when some resource is overused.
On Mon, Mar 18, 2013 at 3:54 PM, David Parks <da...@yahoo.com> wrote:
> I want 20 servers, I got 7, so I want to make the most of the 7 I have. Each
> of the 7 servers have: 24GB of ram, 4TB, and 8 cores.
>
>
>
> Would it be terribly unwise of me to Run such a configuration:
>
>
>
> · Server #1: NameNode + Master + TaskTracker(reduced
> slots)
>
> · Server #2: CheckpointNode(aka Secondary Name Node) +
> TaskTracker(slightly reduced slots)
>
> · Server #3: TaskTracker
>
> · Server #4: TaskTracker
>
> · Server #5: TaskTracker
>
> · Server #6: TaskTracker
>
> · Server #7: TaskTracker
>
>
>
> Did I miss anything? Did I shoot myself in the foot anywhere?
>
>
>
>
--
Harsh J
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Harsh J <ha...@cloudera.com>.
I think its fine to keep it so long as you also monitor the nodes for
issues during heavy load times and tweak slave configs appropriately
when some resource is overused.
On Mon, Mar 18, 2013 at 3:54 PM, David Parks <da...@yahoo.com> wrote:
> I want 20 servers, I got 7, so I want to make the most of the 7 I have. Each
> of the 7 servers have: 24GB of ram, 4TB, and 8 cores.
>
>
>
> Would it be terribly unwise of me to Run such a configuration:
>
>
>
> · Server #1: NameNode + Master + TaskTracker(reduced
> slots)
>
> · Server #2: CheckpointNode(aka Secondary Name Node) +
> TaskTracker(slightly reduced slots)
>
> · Server #3: TaskTracker
>
> · Server #4: TaskTracker
>
> · Server #5: TaskTracker
>
> · Server #6: TaskTracker
>
> · Server #7: TaskTracker
>
>
>
> Did I miss anything? Did I shoot myself in the foot anywhere?
>
>
>
>
--
Harsh J
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Harsh J <ha...@cloudera.com>.
I think its fine to keep it so long as you also monitor the nodes for
issues during heavy load times and tweak slave configs appropriately
when some resource is overused.
On Mon, Mar 18, 2013 at 3:54 PM, David Parks <da...@yahoo.com> wrote:
> I want 20 servers, I got 7, so I want to make the most of the 7 I have. Each
> of the 7 servers have: 24GB of ram, 4TB, and 8 cores.
>
>
>
> Would it be terribly unwise of me to Run such a configuration:
>
>
>
> · Server #1: NameNode + Master + TaskTracker(reduced
> slots)
>
> · Server #2: CheckpointNode(aka Secondary Name Node) +
> TaskTracker(slightly reduced slots)
>
> · Server #3: TaskTracker
>
> · Server #4: TaskTracker
>
> · Server #5: TaskTracker
>
> · Server #6: TaskTracker
>
> · Server #7: TaskTracker
>
>
>
> Did I miss anything? Did I shoot myself in the foot anywhere?
>
>
>
>
--
Harsh J
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Yanbo Liang <ya...@gmail.com>.
I think it is inadvisable to put NameNode and Master(JobTracker) placed in
the same machine,
because the two one are resource intensive applications.
2013/3/18 David Parks <da...@yahoo.com>
> I want 20 servers, I got 7, so I want to make the most of the 7 I have.
> Each of the 7 servers have: 24GB of ram, 4TB, and 8 cores.****
>
> ** **
>
> Would it be terribly unwise of me to Run such a configuration:****
>
> ** **
>
> **· **Server #1: NameNode + Master +
> TaskTracker(reduced slots)****
>
> **· **Server #2: CheckpointNode(aka Secondary Name
> Node) + TaskTracker(slightly reduced slots)****
>
> **· **Server #3: TaskTracker****
>
> **· **Server #4: TaskTracker****
>
> **· **Server #5: TaskTracker****
>
> **· **Server #6: TaskTracker****
>
> **· **Server #7: TaskTracker****
>
> ** **
>
> Did I miss anything? Did I shoot myself in the foot anywhere?****
>
> ** **
>
> ** **
>
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Yanbo Liang <ya...@gmail.com>.
I think it is inadvisable to put NameNode and Master(JobTracker) placed in
the same machine,
because the two one are resource intensive applications.
2013/3/18 David Parks <da...@yahoo.com>
> I want 20 servers, I got 7, so I want to make the most of the 7 I have.
> Each of the 7 servers have: 24GB of ram, 4TB, and 8 cores.****
>
> ** **
>
> Would it be terribly unwise of me to Run such a configuration:****
>
> ** **
>
> **· **Server #1: NameNode + Master +
> TaskTracker(reduced slots)****
>
> **· **Server #2: CheckpointNode(aka Secondary Name
> Node) + TaskTracker(slightly reduced slots)****
>
> **· **Server #3: TaskTracker****
>
> **· **Server #4: TaskTracker****
>
> **· **Server #5: TaskTracker****
>
> **· **Server #6: TaskTracker****
>
> **· **Server #7: TaskTracker****
>
> ** **
>
> Did I miss anything? Did I shoot myself in the foot anywhere?****
>
> ** **
>
> ** **
>
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Jens Scheidtmann <je...@gmail.com>.
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Yanbo Liang <ya...@gmail.com>.
I think it is inadvisable to put NameNode and Master(JobTracker) placed in
the same machine,
because the two one are resource intensive applications.
2013/3/18 David Parks <da...@yahoo.com>
> I want 20 servers, I got 7, so I want to make the most of the 7 I have.
> Each of the 7 servers have: 24GB of ram, 4TB, and 8 cores.****
>
> ** **
>
> Would it be terribly unwise of me to Run such a configuration:****
>
> ** **
>
> **· **Server #1: NameNode + Master +
> TaskTracker(reduced slots)****
>
> **· **Server #2: CheckpointNode(aka Secondary Name
> Node) + TaskTracker(slightly reduced slots)****
>
> **· **Server #3: TaskTracker****
>
> **· **Server #4: TaskTracker****
>
> **· **Server #5: TaskTracker****
>
> **· **Server #6: TaskTracker****
>
> **· **Server #7: TaskTracker****
>
> ** **
>
> Did I miss anything? Did I shoot myself in the foot anywhere?****
>
> ** **
>
> ** **
>
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Jens Scheidtmann <je...@gmail.com>.
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Yanbo Liang <ya...@gmail.com>.
I think it is inadvisable to put NameNode and Master(JobTracker) placed in
the same machine,
because the two one are resource intensive applications.
2013/3/18 David Parks <da...@yahoo.com>
> I want 20 servers, I got 7, so I want to make the most of the 7 I have.
> Each of the 7 servers have: 24GB of ram, 4TB, and 8 cores.****
>
> ** **
>
> Would it be terribly unwise of me to Run such a configuration:****
>
> ** **
>
> **· **Server #1: NameNode + Master +
> TaskTracker(reduced slots)****
>
> **· **Server #2: CheckpointNode(aka Secondary Name
> Node) + TaskTracker(slightly reduced slots)****
>
> **· **Server #3: TaskTracker****
>
> **· **Server #4: TaskTracker****
>
> **· **Server #5: TaskTracker****
>
> **· **Server #6: TaskTracker****
>
> **· **Server #7: TaskTracker****
>
> ** **
>
> Did I miss anything? Did I shoot myself in the foot anywhere?****
>
> ** **
>
> ** **
>
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Jens Scheidtmann <je...@gmail.com>.
David,
You didn't look at how resource intensive your map/reduce jobs are.
Best regards,
Jens
Re: On a small cluster can we double up namenode/master with tasktrackers?
Posted by Harsh J <ha...@cloudera.com>.
I think its fine to keep it so long as you also monitor the nodes for
issues during heavy load times and tweak slave configs appropriately
when some resource is overused.
On Mon, Mar 18, 2013 at 3:54 PM, David Parks <da...@yahoo.com> wrote:
> I want 20 servers, I got 7, so I want to make the most of the 7 I have. Each
> of the 7 servers have: 24GB of ram, 4TB, and 8 cores.
>
>
>
> Would it be terribly unwise of me to Run such a configuration:
>
>
>
> · Server #1: NameNode + Master + TaskTracker(reduced
> slots)
>
> · Server #2: CheckpointNode(aka Secondary Name Node) +
> TaskTracker(slightly reduced slots)
>
> · Server #3: TaskTracker
>
> · Server #4: TaskTracker
>
> · Server #5: TaskTracker
>
> · Server #6: TaskTracker
>
> · Server #7: TaskTracker
>
>
>
> Did I miss anything? Did I shoot myself in the foot anywhere?
>
>
>
>
--
Harsh J