You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2012/12/07 23:09:23 UTC

JobTracker scheduling when there is no locality optimization.

How does the JobTracker schedule / assign tasks to open TaskTrackers when
there is no locality constraints (i.e. when some non file input format,
such as DBInputFormat is used).... Is it random? Or can it optimize for IO?

-- 
Jay Vyas
http://jayunit100.blogspot.com

Re: JobTracker scheduling when there is no locality optimization.

Posted by Harsh J <ha...@cloudera.com>.
It is currently 'random' (based on incoming tasktracker heartbeat order and
availability of slots). MR2 and MR1 with CapacityScheduler can utilize
memory resource request parameters in the job config to schedule based
on memory-constraints, which is the highest you could do at the moment
(CPU based scheduling is arriving soon for YARN, though).

On Sat, Dec 8, 2012 at 3:39 AM, Jay Vyas <ja...@gmail.com> wrote:
> How does the JobTracker schedule / assign tasks to open TaskTrackers when
> there is no locality constraints (i.e. when some non file input format,
> such as DBInputFormat is used).... Is it random? Or can it optimize for IO?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com



--
Harsh J

Re: JobTracker scheduling when there is no locality optimization.

Posted by Harsh J <ha...@cloudera.com>.
It is currently 'random' (based on incoming tasktracker heartbeat order and
availability of slots). MR2 and MR1 with CapacityScheduler can utilize
memory resource request parameters in the job config to schedule based
on memory-constraints, which is the highest you could do at the moment
(CPU based scheduling is arriving soon for YARN, though).

On Sat, Dec 8, 2012 at 3:39 AM, Jay Vyas <ja...@gmail.com> wrote:
> How does the JobTracker schedule / assign tasks to open TaskTrackers when
> there is no locality constraints (i.e. when some non file input format,
> such as DBInputFormat is used).... Is it random? Or can it optimize for IO?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com



--
Harsh J

Re: JobTracker scheduling when there is no locality optimization.

Posted by Harsh J <ha...@cloudera.com>.
It is currently 'random' (based on incoming tasktracker heartbeat order and
availability of slots). MR2 and MR1 with CapacityScheduler can utilize
memory resource request parameters in the job config to schedule based
on memory-constraints, which is the highest you could do at the moment
(CPU based scheduling is arriving soon for YARN, though).

On Sat, Dec 8, 2012 at 3:39 AM, Jay Vyas <ja...@gmail.com> wrote:
> How does the JobTracker schedule / assign tasks to open TaskTrackers when
> there is no locality constraints (i.e. when some non file input format,
> such as DBInputFormat is used).... Is it random? Or can it optimize for IO?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com



--
Harsh J

Re: JobTracker scheduling when there is no locality optimization.

Posted by Harsh J <ha...@cloudera.com>.
It is currently 'random' (based on incoming tasktracker heartbeat order and
availability of slots). MR2 and MR1 with CapacityScheduler can utilize
memory resource request parameters in the job config to schedule based
on memory-constraints, which is the highest you could do at the moment
(CPU based scheduling is arriving soon for YARN, though).

On Sat, Dec 8, 2012 at 3:39 AM, Jay Vyas <ja...@gmail.com> wrote:
> How does the JobTracker schedule / assign tasks to open TaskTrackers when
> there is no locality constraints (i.e. when some non file input format,
> such as DBInputFormat is used).... Is it random? Or can it optimize for IO?
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com



--
Harsh J