You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Nicolae Marasoiu <ni...@adswizz.com> on 2015/11/23 16:58:43 UTC
yarn does not allocate enough tasks/containers to my available node
Hi,
Tasks are allocated to my nodes by memory.
Initially they are allocated ok across the cluster.
After a while, one of the nodes does not receive new tasks fast enough: it gets to 0 tasks and from time to time I see it having 1 task which it finished in seconds.
It is true that I currently have a problem of many small input files.
And probably the fact that the nodes are oversubscribed in cpu by a factor of 2-3 (according to load average) is not helping.
But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node at once (not one by one), and 2. why yarn is slow in allocating tasks? (I understand that allocating a new task/container in a few seconds may/may not be considered slow).
Pls advise,
Nicu
RE: yarn does not allocate enough tasks/containers to my available node
Posted by Bikas Saha <bi...@apache.org>.
Which scheduler is being used? Capacity/Fair/Something else?
From: Nicolae Marasoiu [mailto:nicolae.marasoiu@adswizz.com]
Sent: Monday, November 23, 2015 7:59 AM
To: user@hadoop.apache.org
Subject: yarn does not allocate enough tasks/containers to my available node
Hi,
Tasks are allocated to my nodes by memory.
Initially they are allocated ok across the cluster.
After a while, one of the nodes does not receive new tasks fast enough: it
gets to 0 tasks and from time to time I see it having 1 task which it
finished in seconds.
It is true that I currently have a problem of many small input files.
And probably the fact that the nodes are oversubscribed in cpu by a factor
of 2-3 (according to load average) is not helping.
But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node
at once (not one by one), and 2. why yarn is slow in allocating tasks? (I
understand that allocating a new task/container in a few seconds may/may not
be considered slow).
Pls advise,
Nicu
RE: yarn does not allocate enough tasks/containers to my available node
Posted by Bikas Saha <bi...@apache.org>.
Which scheduler is being used? Capacity/Fair/Something else?
From: Nicolae Marasoiu [mailto:nicolae.marasoiu@adswizz.com]
Sent: Monday, November 23, 2015 7:59 AM
To: user@hadoop.apache.org
Subject: yarn does not allocate enough tasks/containers to my available node
Hi,
Tasks are allocated to my nodes by memory.
Initially they are allocated ok across the cluster.
After a while, one of the nodes does not receive new tasks fast enough: it
gets to 0 tasks and from time to time I see it having 1 task which it
finished in seconds.
It is true that I currently have a problem of many small input files.
And probably the fact that the nodes are oversubscribed in cpu by a factor
of 2-3 (according to load average) is not helping.
But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node
at once (not one by one), and 2. why yarn is slow in allocating tasks? (I
understand that allocating a new task/container in a few seconds may/may not
be considered slow).
Pls advise,
Nicu
RE: yarn does not allocate enough tasks/containers to my available node
Posted by Bikas Saha <bi...@apache.org>.
Which scheduler is being used? Capacity/Fair/Something else?
From: Nicolae Marasoiu [mailto:nicolae.marasoiu@adswizz.com]
Sent: Monday, November 23, 2015 7:59 AM
To: user@hadoop.apache.org
Subject: yarn does not allocate enough tasks/containers to my available node
Hi,
Tasks are allocated to my nodes by memory.
Initially they are allocated ok across the cluster.
After a while, one of the nodes does not receive new tasks fast enough: it
gets to 0 tasks and from time to time I see it having 1 task which it
finished in seconds.
It is true that I currently have a problem of many small input files.
And probably the fact that the nodes are oversubscribed in cpu by a factor
of 2-3 (according to load average) is not helping.
But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node
at once (not one by one), and 2. why yarn is slow in allocating tasks? (I
understand that allocating a new task/container in a few seconds may/may not
be considered slow).
Pls advise,
Nicu
RE: yarn does not allocate enough tasks/containers to my available node
Posted by Bikas Saha <bi...@apache.org>.
Which scheduler is being used? Capacity/Fair/Something else?
From: Nicolae Marasoiu [mailto:nicolae.marasoiu@adswizz.com]
Sent: Monday, November 23, 2015 7:59 AM
To: user@hadoop.apache.org
Subject: yarn does not allocate enough tasks/containers to my available node
Hi,
Tasks are allocated to my nodes by memory.
Initially they are allocated ok across the cluster.
After a while, one of the nodes does not receive new tasks fast enough: it
gets to 0 tasks and from time to time I see it having 1 task which it
finished in seconds.
It is true that I currently have a problem of many small input files.
And probably the fact that the nodes are oversubscribed in cpu by a factor
of 2-3 (according to load average) is not helping.
But 1. why does yarn not able to bulk allocate some 4 tasks on the idle node
at once (not one by one), and 2. why yarn is slow in allocating tasks? (I
understand that allocating a new task/container in a few seconds may/may not
be considered slow).
Pls advise,
Nicu