You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Thiago Vieira <tp...@gmail.com> on 2013/01/18 04:09:38 UTC
Hadoop Scalability
Hello!
Is common to see this sentence: "Hadoop Scales Linearly". But, is there any
performance evaluation to confirm this?
In my evaluations, Hadoop processing capacity scales linearly, but not
proportional to number of nodes, the processing capacity achieved with 20
nodes is not the double of the processing capacity achieved with 10 nodes.
Is there any evaluation about this?
Thank you!
--
Thiago Vieira
Re: Hadoop Scalability
Posted by Ted Dunning <td...@maprtech.com>.
Also, you may have to adjust your algorithms.
For instance, the conventional standard algorithm for SVD is a Lanczos
iterative algorithm. Iteration in Hadoop is death because of job
invocation time ... what you wind up with is an algorithm that will handle
big data but with a slow-down factor that makes a single node perform at
the same level as 100 Hadoop nodes or more. Scaling with iterative
algorithms like this is irrelevant because of the enormous fixed cost.
On the other hand, you can switch to some of the recently developed
stochastic projection algorithms which give a non-iterative algorithm that
requires 4-7 map-reduce steps (depending on which outputs you need). With
these projection algorithms, Hadoop can out-run other techniques even with
quite modest cluster sizes and will scale linearly.
On Thu, Jan 17, 2013 at 9:47 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Hi Thiago,
> Subjectively: there are a number of items to consider to achieve nearly
> linear scaling:
>
>
> - if the work is well balanced among the tasks - no skew
> - No skew in the association of tasks to nodes. Note: this skew
> actually happens by default if the number of tasks is less than the cluster
> capacity of slots. You will notice that on a cluster with 20 nodes, with
> each node set to 20 mapper tasks, if you launch a job with 20 maps it may
> well have all of them running on one node.
> - with higher number of tasks the risk of having stragglers affecting
> overall throughput/performance increases unless speculative execution were
> set properly
> - hadoop configuration settings come under more pressure with more
> - properly tuning the number of mappers and reducers to (a) your node
> and cluster characteristics and (b) the particular tasks has a large impact
> on performance. In my experience the settings are often set too
> conservatively / too low to take advantage of the node and cluster
> resources
>
> So in summary hadoop itself is capable of nearly linear scaling to low
> thousands of nodes, but configuring the cluster to really achieve that
> requires effort.
>
>
> 2013/1/17 Thiago Vieira <tp...@gmail.com>
>
>> Hello!
>>
>> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
>> any performance evaluation to confirm this?
>>
>> In my evaluations, Hadoop processing capacity scales linearly, but not
>> proportional to number of nodes, the processing capacity achieved with 20
>> nodes is not the double of the processing capacity achieved with 10 nodes.
>> Is there any evaluation about this?
>>
>> Thank you!
>>
>> --
>> Thiago Vieira
>>
>
>
Re: Hadoop Scalability
Posted by Ted Dunning <td...@maprtech.com>.
Also, you may have to adjust your algorithms.
For instance, the conventional standard algorithm for SVD is a Lanczos
iterative algorithm. Iteration in Hadoop is death because of job
invocation time ... what you wind up with is an algorithm that will handle
big data but with a slow-down factor that makes a single node perform at
the same level as 100 Hadoop nodes or more. Scaling with iterative
algorithms like this is irrelevant because of the enormous fixed cost.
On the other hand, you can switch to some of the recently developed
stochastic projection algorithms which give a non-iterative algorithm that
requires 4-7 map-reduce steps (depending on which outputs you need). With
these projection algorithms, Hadoop can out-run other techniques even with
quite modest cluster sizes and will scale linearly.
On Thu, Jan 17, 2013 at 9:47 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Hi Thiago,
> Subjectively: there are a number of items to consider to achieve nearly
> linear scaling:
>
>
> - if the work is well balanced among the tasks - no skew
> - No skew in the association of tasks to nodes. Note: this skew
> actually happens by default if the number of tasks is less than the cluster
> capacity of slots. You will notice that on a cluster with 20 nodes, with
> each node set to 20 mapper tasks, if you launch a job with 20 maps it may
> well have all of them running on one node.
> - with higher number of tasks the risk of having stragglers affecting
> overall throughput/performance increases unless speculative execution were
> set properly
> - hadoop configuration settings come under more pressure with more
> - properly tuning the number of mappers and reducers to (a) your node
> and cluster characteristics and (b) the particular tasks has a large impact
> on performance. In my experience the settings are often set too
> conservatively / too low to take advantage of the node and cluster
> resources
>
> So in summary hadoop itself is capable of nearly linear scaling to low
> thousands of nodes, but configuring the cluster to really achieve that
> requires effort.
>
>
> 2013/1/17 Thiago Vieira <tp...@gmail.com>
>
>> Hello!
>>
>> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
>> any performance evaluation to confirm this?
>>
>> In my evaluations, Hadoop processing capacity scales linearly, but not
>> proportional to number of nodes, the processing capacity achieved with 20
>> nodes is not the double of the processing capacity achieved with 10 nodes.
>> Is there any evaluation about this?
>>
>> Thank you!
>>
>> --
>> Thiago Vieira
>>
>
>
Re: Hadoop Scalability
Posted by Ted Dunning <td...@maprtech.com>.
Also, you may have to adjust your algorithms.
For instance, the conventional standard algorithm for SVD is a Lanczos
iterative algorithm. Iteration in Hadoop is death because of job
invocation time ... what you wind up with is an algorithm that will handle
big data but with a slow-down factor that makes a single node perform at
the same level as 100 Hadoop nodes or more. Scaling with iterative
algorithms like this is irrelevant because of the enormous fixed cost.
On the other hand, you can switch to some of the recently developed
stochastic projection algorithms which give a non-iterative algorithm that
requires 4-7 map-reduce steps (depending on which outputs you need). With
these projection algorithms, Hadoop can out-run other techniques even with
quite modest cluster sizes and will scale linearly.
On Thu, Jan 17, 2013 at 9:47 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Hi Thiago,
> Subjectively: there are a number of items to consider to achieve nearly
> linear scaling:
>
>
> - if the work is well balanced among the tasks - no skew
> - No skew in the association of tasks to nodes. Note: this skew
> actually happens by default if the number of tasks is less than the cluster
> capacity of slots. You will notice that on a cluster with 20 nodes, with
> each node set to 20 mapper tasks, if you launch a job with 20 maps it may
> well have all of them running on one node.
> - with higher number of tasks the risk of having stragglers affecting
> overall throughput/performance increases unless speculative execution were
> set properly
> - hadoop configuration settings come under more pressure with more
> - properly tuning the number of mappers and reducers to (a) your node
> and cluster characteristics and (b) the particular tasks has a large impact
> on performance. In my experience the settings are often set too
> conservatively / too low to take advantage of the node and cluster
> resources
>
> So in summary hadoop itself is capable of nearly linear scaling to low
> thousands of nodes, but configuring the cluster to really achieve that
> requires effort.
>
>
> 2013/1/17 Thiago Vieira <tp...@gmail.com>
>
>> Hello!
>>
>> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
>> any performance evaluation to confirm this?
>>
>> In my evaluations, Hadoop processing capacity scales linearly, but not
>> proportional to number of nodes, the processing capacity achieved with 20
>> nodes is not the double of the processing capacity achieved with 10 nodes.
>> Is there any evaluation about this?
>>
>> Thank you!
>>
>> --
>> Thiago Vieira
>>
>
>
Re: Hadoop Scalability
Posted by Ted Dunning <td...@maprtech.com>.
Also, you may have to adjust your algorithms.
For instance, the conventional standard algorithm for SVD is a Lanczos
iterative algorithm. Iteration in Hadoop is death because of job
invocation time ... what you wind up with is an algorithm that will handle
big data but with a slow-down factor that makes a single node perform at
the same level as 100 Hadoop nodes or more. Scaling with iterative
algorithms like this is irrelevant because of the enormous fixed cost.
On the other hand, you can switch to some of the recently developed
stochastic projection algorithms which give a non-iterative algorithm that
requires 4-7 map-reduce steps (depending on which outputs you need). With
these projection algorithms, Hadoop can out-run other techniques even with
quite modest cluster sizes and will scale linearly.
On Thu, Jan 17, 2013 at 9:47 PM, Stephen Boesch <ja...@gmail.com> wrote:
> Hi Thiago,
> Subjectively: there are a number of items to consider to achieve nearly
> linear scaling:
>
>
> - if the work is well balanced among the tasks - no skew
> - No skew in the association of tasks to nodes. Note: this skew
> actually happens by default if the number of tasks is less than the cluster
> capacity of slots. You will notice that on a cluster with 20 nodes, with
> each node set to 20 mapper tasks, if you launch a job with 20 maps it may
> well have all of them running on one node.
> - with higher number of tasks the risk of having stragglers affecting
> overall throughput/performance increases unless speculative execution were
> set properly
> - hadoop configuration settings come under more pressure with more
> - properly tuning the number of mappers and reducers to (a) your node
> and cluster characteristics and (b) the particular tasks has a large impact
> on performance. In my experience the settings are often set too
> conservatively / too low to take advantage of the node and cluster
> resources
>
> So in summary hadoop itself is capable of nearly linear scaling to low
> thousands of nodes, but configuring the cluster to really achieve that
> requires effort.
>
>
> 2013/1/17 Thiago Vieira <tp...@gmail.com>
>
>> Hello!
>>
>> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
>> any performance evaluation to confirm this?
>>
>> In my evaluations, Hadoop processing capacity scales linearly, but not
>> proportional to number of nodes, the processing capacity achieved with 20
>> nodes is not the double of the processing capacity achieved with 10 nodes.
>> Is there any evaluation about this?
>>
>> Thank you!
>>
>> --
>> Thiago Vieira
>>
>
>
Re: Hadoop Scalability
Posted by Stephen Boesch <ja...@gmail.com>.
Hi Thiago,
Subjectively: there are a number of items to consider to achieve nearly
linear scaling:
- if the work is well balanced among the tasks - no skew
- No skew in the association of tasks to nodes. Note: this skew actually
happens by default if the number of tasks is less than the cluster capacity
of slots. You will notice that on a cluster with 20 nodes, with each node
set to 20 mapper tasks, if you launch a job with 20 maps it may well have
all of them running on one node.
- with higher number of tasks the risk of having stragglers affecting
overall throughput/performance increases unless speculative execution were
set properly
- hadoop configuration settings come under more pressure with more
- properly tuning the number of mappers and reducers to (a) your node
and cluster characteristics and (b) the particular tasks has a large impact
on performance. In my experience the settings are often set too
conservatively / too low to take advantage of the node and cluster
resources
So in summary hadoop itself is capable of nearly linear scaling to low
thousands of nodes, but configuring the cluster to really achieve that
requires effort.
2013/1/17 Thiago Vieira <tp...@gmail.com>
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>
Re: Hadoop Scalability
Posted by Arun C Murthy <ac...@hortonworks.com>.
Obviously the algorithm matters, but here are some very old numbers (things today are much better), but you do see the 'linear' scaling with both nodes and datasets:
http://developer.yahoo.com/blogs/hadoop/posts/2009/05/hadoop_sorts_a_petabyte_in_162/
100TB Sort - 97 mins
1000 TB Sort - 975 mins
Arun
On Jan 17, 2013, at 7:09 PM, Thiago Vieira wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not proportional to number of nodes, the processing capacity achieved with 20 nodes is not the double of the processing capacity achieved with 10 nodes. Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Hadoop Scalability
Posted by Arun C Murthy <ac...@hortonworks.com>.
Obviously the algorithm matters, but here are some very old numbers (things today are much better), but you do see the 'linear' scaling with both nodes and datasets:
http://developer.yahoo.com/blogs/hadoop/posts/2009/05/hadoop_sorts_a_petabyte_in_162/
100TB Sort - 97 mins
1000 TB Sort - 975 mins
Arun
On Jan 17, 2013, at 7:09 PM, Thiago Vieira wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not proportional to number of nodes, the processing capacity achieved with 20 nodes is not the double of the processing capacity achieved with 10 nodes. Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Hadoop Scalability
Posted by Arun C Murthy <ac...@hortonworks.com>.
Obviously the algorithm matters, but here are some very old numbers (things today are much better), but you do see the 'linear' scaling with both nodes and datasets:
http://developer.yahoo.com/blogs/hadoop/posts/2009/05/hadoop_sorts_a_petabyte_in_162/
100TB Sort - 97 mins
1000 TB Sort - 975 mins
Arun
On Jan 17, 2013, at 7:09 PM, Thiago Vieira wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not proportional to number of nodes, the processing capacity achieved with 20 nodes is not the double of the processing capacity achieved with 10 nodes. Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Hadoop Scalability
Posted by Arun C Murthy <ac...@hortonworks.com>.
Obviously the algorithm matters, but here are some very old numbers (things today are much better), but you do see the 'linear' scaling with both nodes and datasets:
http://developer.yahoo.com/blogs/hadoop/posts/2009/05/hadoop_sorts_a_petabyte_in_162/
100TB Sort - 97 mins
1000 TB Sort - 975 mins
Arun
On Jan 17, 2013, at 7:09 PM, Thiago Vieira wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not proportional to number of nodes, the processing capacity achieved with 20 nodes is not the double of the processing capacity achieved with 10 nodes. Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Hadoop Scalability
Posted by Stephen Boesch <ja...@gmail.com>.
Hi Thiago,
Subjectively: there are a number of items to consider to achieve nearly
linear scaling:
- if the work is well balanced among the tasks - no skew
- No skew in the association of tasks to nodes. Note: this skew actually
happens by default if the number of tasks is less than the cluster capacity
of slots. You will notice that on a cluster with 20 nodes, with each node
set to 20 mapper tasks, if you launch a job with 20 maps it may well have
all of them running on one node.
- with higher number of tasks the risk of having stragglers affecting
overall throughput/performance increases unless speculative execution were
set properly
- hadoop configuration settings come under more pressure with more
- properly tuning the number of mappers and reducers to (a) your node
and cluster characteristics and (b) the particular tasks has a large impact
on performance. In my experience the settings are often set too
conservatively / too low to take advantage of the node and cluster
resources
So in summary hadoop itself is capable of nearly linear scaling to low
thousands of nodes, but configuring the cluster to really achieve that
requires effort.
2013/1/17 Thiago Vieira <tp...@gmail.com>
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>
Re: Hadoop Scalability
Posted by Jeff Kubina <je...@gmail.com>.
Thiago, when addressing scaling you want to consider whether the algorithm
scales, and if so, if the systems architecture enables the algorithm to
scale, that is, if the algorithm scales on paper, does is also scale on the
hardware?
Algorithms that communicate an amount of data bounded by a constant will
scale on just about any Hadoop cluster, up to a point. At about 4000 nodes
the namenode server may start to become overwhelmed (a bottleneck), and
slow processing down considerably. I think this bottleneck is eliminated in
a not to distance release of the HDFS.
If the amount of data the algorithm communicates is proportional to the
number of processors (map or reduce jobs), than the network bandwidth of
the cluster must increase proportional to the number of processors (since
Hadoop is based on the bulk synchronous
parallel<http://en.wikipedia.org/wiki/Bulk_synchronous_parallel>
model)
to achieve scaling. In such cases a low bandwidth network will impede
scaling. Bryan Duxbury has a nice blog post about networking a Hadoop
cluster here <http://goo.gl/uVeoM>.
More concisely, I would say that "Hadoop scales on clusters with networks
that scale (up to ~4000 nodes)."
--
Jeff Kubina
On Thu, Jan 17, 2013 at 10:09 PM, Thiago Vieira <tp...@gmail.com> wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>
Re: Hadoop Scalability
Posted by Stephen Boesch <ja...@gmail.com>.
Hi Thiago,
Subjectively: there are a number of items to consider to achieve nearly
linear scaling:
- if the work is well balanced among the tasks - no skew
- No skew in the association of tasks to nodes. Note: this skew actually
happens by default if the number of tasks is less than the cluster capacity
of slots. You will notice that on a cluster with 20 nodes, with each node
set to 20 mapper tasks, if you launch a job with 20 maps it may well have
all of them running on one node.
- with higher number of tasks the risk of having stragglers affecting
overall throughput/performance increases unless speculative execution were
set properly
- hadoop configuration settings come under more pressure with more
- properly tuning the number of mappers and reducers to (a) your node
and cluster characteristics and (b) the particular tasks has a large impact
on performance. In my experience the settings are often set too
conservatively / too low to take advantage of the node and cluster
resources
So in summary hadoop itself is capable of nearly linear scaling to low
thousands of nodes, but configuring the cluster to really achieve that
requires effort.
2013/1/17 Thiago Vieira <tp...@gmail.com>
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>
Re: Hadoop Scalability
Posted by Stephen Boesch <ja...@gmail.com>.
Hi Thiago,
Subjectively: there are a number of items to consider to achieve nearly
linear scaling:
- if the work is well balanced among the tasks - no skew
- No skew in the association of tasks to nodes. Note: this skew actually
happens by default if the number of tasks is less than the cluster capacity
of slots. You will notice that on a cluster with 20 nodes, with each node
set to 20 mapper tasks, if you launch a job with 20 maps it may well have
all of them running on one node.
- with higher number of tasks the risk of having stragglers affecting
overall throughput/performance increases unless speculative execution were
set properly
- hadoop configuration settings come under more pressure with more
- properly tuning the number of mappers and reducers to (a) your node
and cluster characteristics and (b) the particular tasks has a large impact
on performance. In my experience the settings are often set too
conservatively / too low to take advantage of the node and cluster
resources
So in summary hadoop itself is capable of nearly linear scaling to low
thousands of nodes, but configuring the cluster to really achieve that
requires effort.
2013/1/17 Thiago Vieira <tp...@gmail.com>
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>
Re: Hadoop Scalability
Posted by Jeff Kubina <je...@gmail.com>.
Thiago, when addressing scaling you want to consider whether the algorithm
scales, and if so, if the systems architecture enables the algorithm to
scale, that is, if the algorithm scales on paper, does is also scale on the
hardware?
Algorithms that communicate an amount of data bounded by a constant will
scale on just about any Hadoop cluster, up to a point. At about 4000 nodes
the namenode server may start to become overwhelmed (a bottleneck), and
slow processing down considerably. I think this bottleneck is eliminated in
a not to distance release of the HDFS.
If the amount of data the algorithm communicates is proportional to the
number of processors (map or reduce jobs), than the network bandwidth of
the cluster must increase proportional to the number of processors (since
Hadoop is based on the bulk synchronous
parallel<http://en.wikipedia.org/wiki/Bulk_synchronous_parallel>
model)
to achieve scaling. In such cases a low bandwidth network will impede
scaling. Bryan Duxbury has a nice blog post about networking a Hadoop
cluster here <http://goo.gl/uVeoM>.
More concisely, I would say that "Hadoop scales on clusters with networks
that scale (up to ~4000 nodes)."
--
Jeff Kubina
On Thu, Jan 17, 2013 at 10:09 PM, Thiago Vieira <tp...@gmail.com> wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>
Re: Hadoop Scalability
Posted by Jeff Kubina <je...@gmail.com>.
Thiago, when addressing scaling you want to consider whether the algorithm
scales, and if so, if the systems architecture enables the algorithm to
scale, that is, if the algorithm scales on paper, does is also scale on the
hardware?
Algorithms that communicate an amount of data bounded by a constant will
scale on just about any Hadoop cluster, up to a point. At about 4000 nodes
the namenode server may start to become overwhelmed (a bottleneck), and
slow processing down considerably. I think this bottleneck is eliminated in
a not to distance release of the HDFS.
If the amount of data the algorithm communicates is proportional to the
number of processors (map or reduce jobs), than the network bandwidth of
the cluster must increase proportional to the number of processors (since
Hadoop is based on the bulk synchronous
parallel<http://en.wikipedia.org/wiki/Bulk_synchronous_parallel>
model)
to achieve scaling. In such cases a low bandwidth network will impede
scaling. Bryan Duxbury has a nice blog post about networking a Hadoop
cluster here <http://goo.gl/uVeoM>.
More concisely, I would say that "Hadoop scales on clusters with networks
that scale (up to ~4000 nodes)."
--
Jeff Kubina
On Thu, Jan 17, 2013 at 10:09 PM, Thiago Vieira <tp...@gmail.com> wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>
Re: Hadoop Scalability
Posted by Jeff Kubina <je...@gmail.com>.
Thiago, when addressing scaling you want to consider whether the algorithm
scales, and if so, if the systems architecture enables the algorithm to
scale, that is, if the algorithm scales on paper, does is also scale on the
hardware?
Algorithms that communicate an amount of data bounded by a constant will
scale on just about any Hadoop cluster, up to a point. At about 4000 nodes
the namenode server may start to become overwhelmed (a bottleneck), and
slow processing down considerably. I think this bottleneck is eliminated in
a not to distance release of the HDFS.
If the amount of data the algorithm communicates is proportional to the
number of processors (map or reduce jobs), than the network bandwidth of
the cluster must increase proportional to the number of processors (since
Hadoop is based on the bulk synchronous
parallel<http://en.wikipedia.org/wiki/Bulk_synchronous_parallel>
model)
to achieve scaling. In such cases a low bandwidth network will impede
scaling. Bryan Duxbury has a nice blog post about networking a Hadoop
cluster here <http://goo.gl/uVeoM>.
More concisely, I would say that "Hadoop scales on clusters with networks
that scale (up to ~4000 nodes)."
--
Jeff Kubina
On Thu, Jan 17, 2013 at 10:09 PM, Thiago Vieira <tp...@gmail.com> wrote:
> Hello!
>
> Is common to see this sentence: "Hadoop Scales Linearly". But, is there
> any performance evaluation to confirm this?
>
> In my evaluations, Hadoop processing capacity scales linearly, but not
> proportional to number of nodes, the processing capacity achieved with 20
> nodes is not the double of the processing capacity achieved with 10 nodes.
> Is there any evaluation about this?
>
> Thank you!
>
> --
> Thiago Vieira
>