You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jameson Li <ho...@gmail.com> on 2013/01/28 16:29:25 UTC
hadoop imbalanced topology
Hi,
Is it right that hadoop Network Topology design treat an imbalanced
topology as an invalid topology?
This is mentioned here
https://issues.apache.org/jira/browse/MAPREDUCE-4095(TestJobInProgress#testLocalityuses
a bogus topology.)
said:"(
https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf)
which does not support imbalanced topology. "
But when I re-read the desigh pdf, there just draw a balanced topology but
really not mentioned it will not support imbalanced topology.
Also I found this JIRA:
https://issues.apache.org/jira/browse/HADOOP-8159(NetworkTopology: getLeaf
should check for invalid topologies).
Just no why for my question, because of the design(Right?):
*Why below is a invalid topology(except imbalanced), for hadoop rack
awareness:*
*serverA /default-rack ##default rack*
*serverB /dc1/rack2 ##actually the node has a pre "dc1"*
*Should all of the hadoop datanode have the same level number? If not,
there will be some issues for namenode and jt(like scheduler deadlock)?*
专注于Mysql,MSSQL,Oracle,Hadoop
Re: hadoop imbalanced topology
Posted by Harsh J <ha...@cloudera.com>.
Hi Jameson,
Inline reply below for a specific point of yours ("scheduler deadlock"):
On Mon, Jan 28, 2013 at 8:59 PM, Jameson Li <ho...@gmail.com> wrote:
> Hi,
>
> Is it right that hadoop Network Topology design treat an imbalanced topology
> as an invalid topology?
>
> This is mentioned here
> https://issues.apache.org/jira/browse/MAPREDUCE-4095(TestJobInProgress#testLocality
> uses a bogus topology.)
> said:"(https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf)
> which does not support imbalanced topology. "
>
> But when I re-read the desigh pdf, there just draw a balanced topology but
> really not mentioned it will not support imbalanced topology.
>
> Also I found this JIRA:
> https://issues.apache.org/jira/browse/HADOOP-8159(NetworkTopology: getLeaf
> should check for invalid topologies).
>
> Just no why for my question, because of the design(Right?):
> Why below is a invalid topology(except imbalanced), for hadoop rack
> awareness:
> serverA /default-rack ##default rack
> serverB /dc1/rack2 ##actually the node has a pre "dc1"
> Should all of the hadoop datanode have the same level number? If not, there
> will be some issues for namenode and jt(like scheduler deadlock)?
This was a problem in earlier releases, where an imbalanced or
improper config for topology levels could lead to a jarring halt in
the JobTracker's schedulers. This has been fixed in 1.1.0 already via
https://issues.apache.org/jira/browse/MAPREDUCE-1740.
--
Harsh J
Re: hadoop imbalanced topology
Posted by Harsh J <ha...@cloudera.com>.
Hi Jameson,
Inline reply below for a specific point of yours ("scheduler deadlock"):
On Mon, Jan 28, 2013 at 8:59 PM, Jameson Li <ho...@gmail.com> wrote:
> Hi,
>
> Is it right that hadoop Network Topology design treat an imbalanced topology
> as an invalid topology?
>
> This is mentioned here
> https://issues.apache.org/jira/browse/MAPREDUCE-4095(TestJobInProgress#testLocality
> uses a bogus topology.)
> said:"(https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf)
> which does not support imbalanced topology. "
>
> But when I re-read the desigh pdf, there just draw a balanced topology but
> really not mentioned it will not support imbalanced topology.
>
> Also I found this JIRA:
> https://issues.apache.org/jira/browse/HADOOP-8159(NetworkTopology: getLeaf
> should check for invalid topologies).
>
> Just no why for my question, because of the design(Right?):
> Why below is a invalid topology(except imbalanced), for hadoop rack
> awareness:
> serverA /default-rack ##default rack
> serverB /dc1/rack2 ##actually the node has a pre "dc1"
> Should all of the hadoop datanode have the same level number? If not, there
> will be some issues for namenode and jt(like scheduler deadlock)?
This was a problem in earlier releases, where an imbalanced or
improper config for topology levels could lead to a jarring halt in
the JobTracker's schedulers. This has been fixed in 1.1.0 already via
https://issues.apache.org/jira/browse/MAPREDUCE-1740.
--
Harsh J
Re: hadoop imbalanced topology
Posted by Harsh J <ha...@cloudera.com>.
Hi Jameson,
Inline reply below for a specific point of yours ("scheduler deadlock"):
On Mon, Jan 28, 2013 at 8:59 PM, Jameson Li <ho...@gmail.com> wrote:
> Hi,
>
> Is it right that hadoop Network Topology design treat an imbalanced topology
> as an invalid topology?
>
> This is mentioned here
> https://issues.apache.org/jira/browse/MAPREDUCE-4095(TestJobInProgress#testLocality
> uses a bogus topology.)
> said:"(https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf)
> which does not support imbalanced topology. "
>
> But when I re-read the desigh pdf, there just draw a balanced topology but
> really not mentioned it will not support imbalanced topology.
>
> Also I found this JIRA:
> https://issues.apache.org/jira/browse/HADOOP-8159(NetworkTopology: getLeaf
> should check for invalid topologies).
>
> Just no why for my question, because of the design(Right?):
> Why below is a invalid topology(except imbalanced), for hadoop rack
> awareness:
> serverA /default-rack ##default rack
> serverB /dc1/rack2 ##actually the node has a pre "dc1"
> Should all of the hadoop datanode have the same level number? If not, there
> will be some issues for namenode and jt(like scheduler deadlock)?
This was a problem in earlier releases, where an imbalanced or
improper config for topology levels could lead to a jarring halt in
the JobTracker's schedulers. This has been fixed in 1.1.0 already via
https://issues.apache.org/jira/browse/MAPREDUCE-1740.
--
Harsh J
Re: hadoop imbalanced topology
Posted by Harsh J <ha...@cloudera.com>.
Hi Jameson,
Inline reply below for a specific point of yours ("scheduler deadlock"):
On Mon, Jan 28, 2013 at 8:59 PM, Jameson Li <ho...@gmail.com> wrote:
> Hi,
>
> Is it right that hadoop Network Topology design treat an imbalanced topology
> as an invalid topology?
>
> This is mentioned here
> https://issues.apache.org/jira/browse/MAPREDUCE-4095(TestJobInProgress#testLocality
> uses a bogus topology.)
> said:"(https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf)
> which does not support imbalanced topology. "
>
> But when I re-read the desigh pdf, there just draw a balanced topology but
> really not mentioned it will not support imbalanced topology.
>
> Also I found this JIRA:
> https://issues.apache.org/jira/browse/HADOOP-8159(NetworkTopology: getLeaf
> should check for invalid topologies).
>
> Just no why for my question, because of the design(Right?):
> Why below is a invalid topology(except imbalanced), for hadoop rack
> awareness:
> serverA /default-rack ##default rack
> serverB /dc1/rack2 ##actually the node has a pre "dc1"
> Should all of the hadoop datanode have the same level number? If not, there
> will be some issues for namenode and jt(like scheduler deadlock)?
This was a problem in earlier releases, where an imbalanced or
improper config for topology levels could lead to a jarring halt in
the JobTracker's schedulers. This has been fixed in 1.1.0 already via
https://issues.apache.org/jira/browse/MAPREDUCE-1740.
--
Harsh J