You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Костарев А.Ф." <ka...@ics.perm.ru> on 2013/07/09 11:48:49 UTC
Algorithm of distribution Map and Reduce tasks at various topology
of a network
Hi
I have claster in two datacenters
CLUSTER
|
+--------+---------+
| |
datacenter1 datacenter2
| |
rack1 rack1
| | |
+-a1 | +-b1
| | |
+-a2 | +-b3
|
rack2
+-b3
Cluster have file with repcica coefficient=5
All files's blocks resides on all servers of cluser.
When I work with standart MapReduce (MRv1) (called on b1) Map and
Rediuce task runs on all servers b1, b2, b3, a1, a2
When I work with YARN (MRv2) (called on b1) Map and Reduce task runs
only on b1, b2, b3
Can I run in YARN Map tasks on all servers?
--
Консультант 1-й категории
Костарев А.Ф.
RE: Algorithm of distribution Map and Reduce tasks at various
topology of a network
Posted by Bikas Saha <bi...@hortonworks.com>.
Can you please share ResourceManager logs, the application id for the job,
the cluster machine configuration.
Thanks
Bikas
-----Original Message-----
From: Jun Ping Du [mailto:jdu@vmware.com]
Sent: Tuesday, July 16, 2013 6:15 AM
To: Костарев А.Ф.
Cc: yarn-dev@hadoop.apache.org
Subject: Re: Algorithm of distribution Map and Reduce tasks at various
topology of a network
Hi Костарев,
I don't think this issue relates to federation. It looks like a1 and a2
doesn't register to RM successfully. It would helpful if you can share
NodeManager's log to see what's wrong with a1 and a2 nodes in registration.
Thanks,
Junping
----- Original Message -----
From: "Костарев А.Ф." <ka...@ics.perm.ru>
To: "Jun Ping Du" <jd...@vmware.com>
Cc: yarn-dev@hadoop.apache.org
Sent: Tuesday, July 16, 2013 6:43:45 PM
Subject: Re: Algorithm of distribution Map and Reduce tasks at various
topology of a network
Hi Jun Ping Du!
Do You use fedaration?
Our cluster works in federation mode. Today we launched tests again. Input
file had size 600 Mb and size of block 6. So, there were 100 map tasks.
And we saw the same result: all map tasks were executed within one
datacenter.
In web-interfaces of HDFS and YARN we saw different information. Page of
HDFS contained information, that there were 6 nodes in cluster, page of YARN
said only ablout 3 nodes.
Screenshots and configuration files are attached.
What settings we need to change to execute tasks on all nodes?
On 07/15/2013 05:52 PM, Jun Ping Du wrote:
Hi Костарев,
I tried to reproduce your case on my 5-nodes setup (with 2 nodes in
dc1/rack1, 1 node in dc1/rack2 and 2 nodes in dc2/rack2) but didn't see
anything unusual. In my test, even with 3 replicas, I saw job with 150 map
tasks distributed across all nodes no matter what datacenter is.
Can you try again with a job with more map tasks as it is pretty random in
scheduling if your job only have 10 map tasks. In your case, it seems b1, b3
and b2 take 3-4 maps away in one heartbeat which is pretty normal case. Let
me know the distribution and version you are using if it still not work with
more tasks.
btw, you can find history log for each job in application page. Isn't it?
Thanks,
Junping
--
Консультант 1-й категории
Костарев А.Ф.
Re: Algorithm of distribution Map and Reduce tasks at various
topology of a network
Posted by Jun Ping Du <jd...@vmware.com>.
Hi Костарев,
I don't think this issue relates to federation. It looks like a1 and a2 doesn't register to RM successfully. It would helpful if you can share NodeManager's log to see what's wrong with a1 and a2 nodes in registration.
Thanks,
Junping
----- Original Message -----
From: "Костарев А.Ф." <ka...@ics.perm.ru>
To: "Jun Ping Du" <jd...@vmware.com>
Cc: yarn-dev@hadoop.apache.org
Sent: Tuesday, July 16, 2013 6:43:45 PM
Subject: Re: Algorithm of distribution Map and Reduce tasks at various topology of a network
Hi Jun Ping Du!
Do You use fedaration?
Our cluster works in federation mode. Today we launched tests again. Input file had size 600 Mb and size of block 6. So, there were 100 map tasks.
And we saw the same result: all map tasks were executed within one datacenter.
In web-interfaces of HDFS and YARN we saw different information. Page of HDFS contained information, that there were 6 nodes in cluster, page of YARN said only ablout 3 nodes.
Screenshots and configuration files are attached.
What settings we need to change to execute tasks on all nodes?
On 07/15/2013 05:52 PM, Jun Ping Du wrote:
Hi Костарев,
I tried to reproduce your case on my 5-nodes setup (with 2 nodes in dc1/rack1, 1 node in dc1/rack2 and 2 nodes in dc2/rack2) but didn't see anything unusual. In my test, even with 3 replicas, I saw job with 150 map tasks distributed across all nodes no matter what datacenter is.
Can you try again with a job with more map tasks as it is pretty random in scheduling if your job only have 10 map tasks. In your case, it seems b1, b3 and b2 take 3-4 maps away in one heartbeat which is pretty normal case. Let me know the distribution and version you are using if it still not work with more tasks.
btw, you can find history log for each job in application page. Isn't it?
Thanks,
Junping
--
Консультант 1-й категории
Костарев А.Ф.
Re: Algorithm of distribution Map and Reduce tasks at various topology
of a network
Posted by "Костарев А.Ф." <ka...@ics.perm.ru>.
Hi Jun Ping Du!
Do You use fedaration?
Our cluster works in *_federation_* mode. Today we launched tests again.
Input file had size 600 Mb and size of block 6. So, there were 100 map
tasks.
And we saw the same result: all map tasks were executed within one
datacenter.
In web-interfaces of HDFS and YARN we saw different information. Page of
HDFS contained information, that there were 6 nodes in cluster, page of
YARN said only ablout 3 nodes.
Screenshots and configuration files are attached.
What settings we need to change to execute tasks on all nodes?
On 07/15/2013 05:52 PM, Jun Ping Du wrote:
> Hi Костарев,
> I tried to reproduce your case on my 5-nodes setup (with 2 nodes
> in dc1/rack1, 1 node in dc1/rack2 and 2 nodes in dc2/rack2) but didn't
> see anything unusual. In my test, even with 3 replicas, I saw job with
> 150 map tasks distributed across all nodes no matter what datacenter is.
> Can you try again with a job with more map tasks as it is pretty
> random in scheduling if your job only have 10 map tasks. In your case,
> it seems b1, b3 and b2 take 3-4 maps away in one heartbeat which is
> pretty normal case. Let me know the distribution and version you are
> using if it still not work with more tasks.
> btw, you can find history log for each job in application page.
> Isn't it?
>
> Thanks,
>
> Junping
--
Консультант 1-й категории
Костарев А.Ф.
Re: Algorithm of distribution Map and Reduce tasks at various
topology of a network
Posted by Jun Ping Du <jd...@vmware.com>.
Hi Костарев,
I tried to reproduce your case on my 5-nodes setup (with 2 nodes in dc1/rack1, 1 node in dc1/rack2 and 2 nodes in dc2/rack2) but didn't see anything unusual. In my test, even with 3 replicas, I saw job with 150 map tasks distributed across all nodes no matter what datacenter is.
Can you try again with a job with more map tasks as it is pretty random in scheduling if your job only have 10 map tasks. In your case, it seems b1, b3 and b2 take 3-4 maps away in one heartbeat which is pretty normal case. Let me know the distribution and version you are using if it still not work with more tasks.
btw, you can find history log for each job in application page. Isn't it?
Thanks,
Junping
----- Original Message -----
From: "Костарев А.Ф." <ka...@ics.perm.ru>
To: yarn-dev@hadoop.apache.org, jdu@vmware.com
Sent: Friday, July 12, 2013 11:42:14 AM
Subject: Re: Algorithm of distribution Map and Reduce tasks at various topology of a network
On 07/09/2013 05:36 PM, "Костарев А.Ф." wrote:
Hi Junping!
We have launched MapReduce tasks in YARN cluster. Its topology is described in file topology.data. All files are here: https://drive.google.com/folderview?id=0B1dEBvIR3qbpeFVxR0R6UXhyNEU&usp=sharing
And we found difference between MRv1 and YARN. In usual MapReduce tasks were executed everywhere, but in YARN they are executed only within one datacenter. It's shown on screenshots. Input file had replication factor 5.
So, is it possible to run 1 job not only in 1 datacenter but on all servers of cluster at the same time?
One more question. In MapReduce in ouptut directory in HDFS there were: _SUCCESS, log and part-r-xxxx. Now i can't find log. Is it in another place or in YARN there isn't such a file?
Thank you for your prompt response
We will try to repeat the test on Thursday and show more details
On 07/09/2013 05:18 PM, Jun Ping Du wrote:
<blockquote>
Hi Костарев,
I think it should work for YARN even YARN doesn't support layer above rack (actually I am working on supporting more layers topology for YARN at YARN-18) now.
Current YARN should just recognize your topology as three racks: "dc1/rack1", "dc2/rack1", "dc2/rack2". Each node (NM) with free resources should be assigned with containers in heartbeat with RM no matter what locality level there. The only exception case should be: 1. no pending resource requests 2. NM capacity is too small to meet resource request 3. delay scheduling is enabled and no data-local attempt. In your case, I don't see anything stop task assignment on a1 and a2. Anyone here can correct me if any misunderstanding here. :)
Anyway, I will give it a try (as your configuration) later to see if some bugs in boundary cases there or it could be some misconfiguration. Which minor version (2.0.x or trunk) you are using now?
Thanks,
Junping
----- Original Message -----
From: "Костарев А.Ф." <ka...@ics.perm.ru>
To: yarn-dev@hadoop.apache.org
Sent: Tuesday, July 9, 2013 5:48:49 PM
Subject: Algorithm of distribution Map and Reduce tasks at various topology of a network
Hi
I have claster in two datacenters
CLUSTER
|
+--------+---------+
| |
datacenter1 datacenter2
| |
rack1 rack1
| | |
+-a1 | +-b1
| | |
+-a2 | +-b3
|
rack2
+-b3
Cluster have file with repcica coefficient=5
All files's blocks resides on all servers of cluser.
When I work with standart MapReduce (MRv1) (called on b1) Map and
Rediuce task runs on all servers b1, b2, b3, a1, a2
When I work with YARN (MRv2) (called on b1) Map and Reduce task runs
only on b1, b2, b3
Can I run in YARN Map tasks on all servers?
</blockquote>
--
Консультант 1-й категории
Костарев А.Ф.
Re: Algorithm of distribution Map and Reduce tasks at various topology
of a network
Posted by "Костарев А.Ф." <ka...@ics.perm.ru>.
On 07/09/2013 05:36 PM, "Костарев А.Ф." wrote:
Hi Junping!
We have launched MapReduce tasks in YARN cluster. Its topology is
described in file topology.data. All files are here:
https://drive.google.com/folderview?id=0B1dEBvIR3qbpeFVxR0R6UXhyNEU&usp=sharing
And we found difference between MRv1 and YARN. In usual MapReduce tasks
were executed everywhere, but in YARN they are executed only within one
datacenter. It's shown on screenshots. Input file had replication factor 5.
So, is it possible to run 1 job not only in 1 datacenter but on all
servers of cluster at the same time?
One more question. In MapReduce in ouptut directory in HDFS there were:
_SUCCESS, log and part-r-xxxx. Now i can't find log. Is it in another
place or in YARN there isn't such a file?
> Thank you for your prompt response
>
> We will try to repeat the test on Thursday and show more details
>
>
> On 07/09/2013 05:18 PM, Jun Ping Du wrote:
>> Hi Костарев,
>> I think it should work for YARN even YARN doesn't support layer
>> above rack (actually I am working on supporting more layers topology
>> for YARN at YARN-18) now.
>> Current YARN should just recognize your topology as three racks:
>> "dc1/rack1", "dc2/rack1", "dc2/rack2". Each node (NM) with free
>> resources should be assigned with containers in heartbeat with RM no
>> matter what locality level there. The only exception case should be:
>> 1. no pending resource requests 2. NM capacity is too small to meet
>> resource request 3. delay scheduling is enabled and no data-local
>> attempt. In your case, I don't see anything stop task assignment on
>> a1 and a2. Anyone here can correct me if any misunderstanding here. :)
>> Anyway, I will give it a try (as your configuration) later to see
>> if some bugs in boundary cases there or it could be some
>> misconfiguration. Which minor version (2.0.x or trunk) you are using
>> now?
>>
>> Thanks,
>>
>> Junping
>>
>> ----- Original Message -----
>> From: "Костарев А.Ф." <ka...@ics.perm.ru>
>> To: yarn-dev@hadoop.apache.org
>> Sent: Tuesday, July 9, 2013 5:48:49 PM
>> Subject: Algorithm of distribution Map and Reduce tasks at various
>> topology of a network
>>
>> Hi
>> I have claster in two datacenters
>>
>> CLUSTER
>> |
>> +--------+---------+
>> | |
>> datacenter1 datacenter2
>> | |
>> rack1 rack1
>> | | |
>> +-a1 | +-b1
>> | | |
>> +-a2 | +-b3
>> |
>> rack2
>> +-b3
>>
>>
>> Cluster have file with repcica coefficient=5
>> All files's blocks resides on all servers of cluser.
>>
>> When I work with standart MapReduce (MRv1) (called on b1) Map and
>> Rediuce task runs on all servers b1, b2, b3, a1, a2
>> When I work with YARN (MRv2) (called on b1) Map and Reduce task runs
>> only on b1, b2, b3
>>
>> Can I run in YARN Map tasks on all servers?
>>
>>
>
>
--
Консультант 1-й категории
Костарев А.Ф.
Re: Algorithm of distribution Map and Reduce tasks at various topology
of a network
Posted by "Костарев А.Ф." <ka...@ics.perm.ru>.
Hi Junping
Thank you for your prompt response
We will try to repeat the test on Thursday and show more details
On 07/09/2013 05:18 PM, Jun Ping Du wrote:
> Hi Костарев,
> I think it should work for YARN even YARN doesn't support layer above rack (actually I am working on supporting more layers topology for YARN at YARN-18) now.
> Current YARN should just recognize your topology as three racks: "dc1/rack1", "dc2/rack1", "dc2/rack2". Each node (NM) with free resources should be assigned with containers in heartbeat with RM no matter what locality level there. The only exception case should be: 1. no pending resource requests 2. NM capacity is too small to meet resource request 3. delay scheduling is enabled and no data-local attempt. In your case, I don't see anything stop task assignment on a1 and a2. Anyone here can correct me if any misunderstanding here. :)
> Anyway, I will give it a try (as your configuration) later to see if some bugs in boundary cases there or it could be some misconfiguration. Which minor version (2.0.x or trunk) you are using now?
>
> Thanks,
>
> Junping
>
> ----- Original Message -----
> From: "Костарев А.Ф." <ka...@ics.perm.ru>
> To: yarn-dev@hadoop.apache.org
> Sent: Tuesday, July 9, 2013 5:48:49 PM
> Subject: Algorithm of distribution Map and Reduce tasks at various topology of a network
>
> Hi
> I have claster in two datacenters
>
> CLUSTER
> |
> +--------+---------+
> | |
> datacenter1 datacenter2
> | |
> rack1 rack1
> | | |
> +-a1 | +-b1
> | | |
> +-a2 | +-b3
> |
> rack2
> +-b3
>
>
> Cluster have file with repcica coefficient=5
> All files's blocks resides on all servers of cluser.
>
> When I work with standart MapReduce (MRv1) (called on b1) Map and
> Rediuce task runs on all servers b1, b2, b3, a1, a2
> When I work with YARN (MRv2) (called on b1) Map and Reduce task runs
> only on b1, b2, b3
>
> Can I run in YARN Map tasks on all servers?
>
>
--
Консультант 1-й категории
Костарев А.Ф.
Re: Algorithm of distribution Map and Reduce tasks at various
topology of a network
Posted by Jun Ping Du <jd...@vmware.com>.
Hi Костарев,
I think it should work for YARN even YARN doesn't support layer above rack (actually I am working on supporting more layers topology for YARN at YARN-18) now.
Current YARN should just recognize your topology as three racks: "dc1/rack1", "dc2/rack1", "dc2/rack2". Each node (NM) with free resources should be assigned with containers in heartbeat with RM no matter what locality level there. The only exception case should be: 1. no pending resource requests 2. NM capacity is too small to meet resource request 3. delay scheduling is enabled and no data-local attempt. In your case, I don't see anything stop task assignment on a1 and a2. Anyone here can correct me if any misunderstanding here. :)
Anyway, I will give it a try (as your configuration) later to see if some bugs in boundary cases there or it could be some misconfiguration. Which minor version (2.0.x or trunk) you are using now?
Thanks,
Junping
----- Original Message -----
From: "Костарев А.Ф." <ka...@ics.perm.ru>
To: yarn-dev@hadoop.apache.org
Sent: Tuesday, July 9, 2013 5:48:49 PM
Subject: Algorithm of distribution Map and Reduce tasks at various topology of a network
Hi
I have claster in two datacenters
CLUSTER
|
+--------+---------+
| |
datacenter1 datacenter2
| |
rack1 rack1
| | |
+-a1 | +-b1
| | |
+-a2 | +-b3
|
rack2
+-b3
Cluster have file with repcica coefficient=5
All files's blocks resides on all servers of cluser.
When I work with standart MapReduce (MRv1) (called on b1) Map and
Rediuce task runs on all servers b1, b2, b3, a1, a2
When I work with YARN (MRv2) (called on b1) Map and Reduce task runs
only on b1, b2, b3
Can I run in YARN Map tasks on all servers?
--
Консультант 1-й категории
Костарев А.Ф