You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Zhu Huijun <na...@gmail.com> on 2008/02/13 17:21:10 UTC
About the support of MapReduce task scheduling schemes
Dear all,
Recently we are invesgating dynamic scheduling of parallel IO-intensive
applications in large-scale cluster. We are interested in Hadoop, especially
its task scheduling schemes. Is the scheduling module a part of any library
of Hadoop, or is it a standalone library? Is there any publications specific
on scheduling in Hadoop? Could you please share some details about
scheduling or suggest some literature of Hadoop?
Thanks!
Best wishes,
Huijun Zhu
Re: About the support of MapReduce task scheduling schemes
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Feb 13, 2008, at 9:39 AM, Jaideep Dhok wrote:
> Hi,
> AFAIK right now they are only doing FCFS scheduling. You can read
> the code
> in org.apache.hadoop.mapred.JobTracker.java. I think the code in
> "getNewTaskForTaskTracker" method.
It actually prefers a host where the data is local. On large jobs, we
get 95% of the maps running locally (assuming hdfs cluster == map/
reduce cluster). You also adding rack locality into the scheduling in
HADOOP-1985.
-- Owen
Re: About the support of MapReduce task scheduling schemes
Posted by Jaideep Dhok <ja...@gmail.com>.
Hi,
AFAIK right now they are only doing FCFS scheduling. You can read the code
in org.apache.hadoop.mapred.JobTracker.java. I think the code in
"getNewTaskForTaskTracker" method.
- Jaideep
On Feb 13, 2008 11:02 PM, Zhu Huijun <na...@gmail.com> wrote:
> Thanks. Do you have any idea about where is the scheduling module in the
> source files?
>
> 2008/2/13, Ahmad Humayun <ah...@gmail.com>:
> >
> > Its not a separate module, as far as I know, its just part of the whole
> > Hadoop implementation. If you would like to use it somewhere else, it
> > would
> > take some serious surgery :)
> >
> > I'm sure you didn't want to hear this, but this is what I know. Although
> > I'll recommend you to ask others too :)
> >
> > regards,
> >
> > On Feb 13, 2008 10:18 PM, Zhu Huijun <na...@gmail.com> wrote:
> >
> > > Thank you, Ahmad Humayun. I am not asking for the idea of MapReduce.
> > What
> > > I
> > > am asking is the scheduling scheme in Hadoop. I am wondering whether
> the
> > > scheduling module is a part of any library implemented in Hadoop, or
> the
> > > scheduling of Hadoop is based on the infrastructure under which Hadoop
> > is
> > > installed.
> > >
> > > 2008/2/13, Ahmad Humayun <ah...@gmail.com>:
> > > >
> > > > The ideas were originally published as MapReduce by Jeffrey
> > > > Dean<http://labs.google.com/people/jeff>and Sanjay
> > > > Ghemawat <http://labs.google.com/people/sanjay> at Google. Have a
> look
> > > at:
> > > > http://labs.google.com/papers/mapreduce.html
> > > >
> > > >
> > > > regards,
> > > >
> > > > On Feb 13, 2008 9:21 PM, Zhu Huijun <na...@gmail.com> wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > Recently we are invesgating dynamic scheduling of parallel
> > > IO-intensive
> > > > > applications in large-scale cluster. We are interested in Hadoop,
> > > > > especially
> > > > > its task scheduling schemes. Is the scheduling module a part of
> any
> > > > > library
> > > > > of Hadoop, or is it a standalone library? Is there any
> publications
> > > > > specific
> > > > > on scheduling in Hadoop? Could you please share some details about
> > > > > scheduling or suggest some literature of Hadoop?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Best wishes,
> > > > >
> > > > > Huijun Zhu
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Ahmad Humayun
> > > > Research Assistant
> > > > Computer Science Dpt., LUMS
> > > > +92 321 4457315
> > > >
> > >
> >
> >
> >
> > --
> > Ahmad Humayun
> > Research Assistant
> > Computer Science Dpt., LUMS
> > +92 321 4457315
> >
>
--
Jaideep Dhok
Re: About the support of MapReduce task scheduling schemes
Posted by Ahmad Humayun <ah...@gmail.com>.
As far as I understand, since this is a distributed computation, you wouldnt
find scheduling at one place. Everything is running as concurrent threads on
separate machines. But you might want to look at the JobTracker (the Master
process) ... you might want to look at its sub-class JobInitThread and see
what things are being run from there on, especially
JobInProgress::initTasks(). Also look at the TaskTracker which manages the
running of all Map and Reduce functions at a specific machine.
hope this helps,
On Feb 13, 2008 10:32 PM, Zhu Huijun <na...@gmail.com> wrote:
> Thanks. Do you have any idea about where is the scheduling module in the
> source files?
>
> 2008/2/13, Ahmad Humayun <ah...@gmail.com>:
> >
> > Its not a separate module, as far as I know, its just part of the whole
> > Hadoop implementation. If you would like to use it somewhere else, it
> > would
> > take some serious surgery :)
> >
> > I'm sure you didn't want to hear this, but this is what I know. Although
> > I'll recommend you to ask others too :)
> >
> > regards,
> >
> > On Feb 13, 2008 10:18 PM, Zhu Huijun <na...@gmail.com> wrote:
> >
> > > Thank you, Ahmad Humayun. I am not asking for the idea of MapReduce.
> > What
> > > I
> > > am asking is the scheduling scheme in Hadoop. I am wondering whether
> the
> > > scheduling module is a part of any library implemented in Hadoop, or
> the
> > > scheduling of Hadoop is based on the infrastructure under which Hadoop
> > is
> > > installed.
> > >
> > > 2008/2/13, Ahmad Humayun <ah...@gmail.com>:
> > > >
> > > > The ideas were originally published as MapReduce by Jeffrey
> > > > Dean<http://labs.google.com/people/jeff>and Sanjay
> > > > Ghemawat <http://labs.google.com/people/sanjay> at Google. Have a
> look
> > > at:
> > > > http://labs.google.com/papers/mapreduce.html
> > > >
> > > >
> > > > regards,
> > > >
> > > > On Feb 13, 2008 9:21 PM, Zhu Huijun <na...@gmail.com> wrote:
> > > >
> > > > > Dear all,
> > > > >
> > > > > Recently we are invesgating dynamic scheduling of parallel
> > > IO-intensive
> > > > > applications in large-scale cluster. We are interested in Hadoop,
> > > > > especially
> > > > > its task scheduling schemes. Is the scheduling module a part of
> any
> > > > > library
> > > > > of Hadoop, or is it a standalone library? Is there any
> publications
> > > > > specific
> > > > > on scheduling in Hadoop? Could you please share some details about
> > > > > scheduling or suggest some literature of Hadoop?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Best wishes,
> > > > >
> > > > > Huijun Zhu
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Ahmad Humayun
> > > > Research Assistant
> > > > Computer Science Dpt., LUMS
> > > > +92 321 4457315
> > > >
> > >
> >
> >
> >
> > --
> > Ahmad Humayun
> > Research Assistant
> > Computer Science Dpt., LUMS
> > +92 321 4457315
> >
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315
Re: About the support of MapReduce task scheduling schemes
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Feb 13, 2008, at 9:32 AM, Zhu Huijun wrote:
> Thanks. Do you have any idea about where is the scheduling module
> in the
> source files?
The scheduling is all done in the JobTracker. The relevant classes
are mostly JobTracker and JobInProgress in org.apache.hadoop.mapred.
You should note that we are in the middle of redesigning them. The
relevant bugs are:
https://issues.apache.org/jira/browse/HADOOP-2119
https://issues.apache.org/jira/browse/HADOOP-2014
-- Owen
Re: About the support of MapReduce task scheduling schemes
Posted by Zhu Huijun <na...@gmail.com>.
Thanks. Do you have any idea about where is the scheduling module in the
source files?
2008/2/13, Ahmad Humayun <ah...@gmail.com>:
>
> Its not a separate module, as far as I know, its just part of the whole
> Hadoop implementation. If you would like to use it somewhere else, it
> would
> take some serious surgery :)
>
> I'm sure you didn't want to hear this, but this is what I know. Although
> I'll recommend you to ask others too :)
>
> regards,
>
> On Feb 13, 2008 10:18 PM, Zhu Huijun <na...@gmail.com> wrote:
>
> > Thank you, Ahmad Humayun. I am not asking for the idea of MapReduce.
> What
> > I
> > am asking is the scheduling scheme in Hadoop. I am wondering whether the
> > scheduling module is a part of any library implemented in Hadoop, or the
> > scheduling of Hadoop is based on the infrastructure under which Hadoop
> is
> > installed.
> >
> > 2008/2/13, Ahmad Humayun <ah...@gmail.com>:
> > >
> > > The ideas were originally published as MapReduce by Jeffrey
> > > Dean<http://labs.google.com/people/jeff>and Sanjay
> > > Ghemawat <http://labs.google.com/people/sanjay> at Google. Have a look
> > at:
> > > http://labs.google.com/papers/mapreduce.html
> > >
> > >
> > > regards,
> > >
> > > On Feb 13, 2008 9:21 PM, Zhu Huijun <na...@gmail.com> wrote:
> > >
> > > > Dear all,
> > > >
> > > > Recently we are invesgating dynamic scheduling of parallel
> > IO-intensive
> > > > applications in large-scale cluster. We are interested in Hadoop,
> > > > especially
> > > > its task scheduling schemes. Is the scheduling module a part of any
> > > > library
> > > > of Hadoop, or is it a standalone library? Is there any publications
> > > > specific
> > > > on scheduling in Hadoop? Could you please share some details about
> > > > scheduling or suggest some literature of Hadoop?
> > > >
> > > > Thanks!
> > > >
> > > > Best wishes,
> > > >
> > > > Huijun Zhu
> > > >
> > >
> > >
> > >
> > > --
> > > Ahmad Humayun
> > > Research Assistant
> > > Computer Science Dpt., LUMS
> > > +92 321 4457315
> > >
> >
>
>
>
> --
> Ahmad Humayun
> Research Assistant
> Computer Science Dpt., LUMS
> +92 321 4457315
>
Re: About the support of MapReduce task scheduling schemes
Posted by Ahmad Humayun <ah...@gmail.com>.
Its not a separate module, as far as I know, its just part of the whole
Hadoop implementation. If you would like to use it somewhere else, it would
take some serious surgery :)
I'm sure you didn't want to hear this, but this is what I know. Although
I'll recommend you to ask others too :)
regards,
On Feb 13, 2008 10:18 PM, Zhu Huijun <na...@gmail.com> wrote:
> Thank you, Ahmad Humayun. I am not asking for the idea of MapReduce. What
> I
> am asking is the scheduling scheme in Hadoop. I am wondering whether the
> scheduling module is a part of any library implemented in Hadoop, or the
> scheduling of Hadoop is based on the infrastructure under which Hadoop is
> installed.
>
> 2008/2/13, Ahmad Humayun <ah...@gmail.com>:
> >
> > The ideas were originally published as MapReduce by Jeffrey
> > Dean<http://labs.google.com/people/jeff>and Sanjay
> > Ghemawat <http://labs.google.com/people/sanjay> at Google. Have a look
> at:
> > http://labs.google.com/papers/mapreduce.html
> >
> >
> > regards,
> >
> > On Feb 13, 2008 9:21 PM, Zhu Huijun <na...@gmail.com> wrote:
> >
> > > Dear all,
> > >
> > > Recently we are invesgating dynamic scheduling of parallel
> IO-intensive
> > > applications in large-scale cluster. We are interested in Hadoop,
> > > especially
> > > its task scheduling schemes. Is the scheduling module a part of any
> > > library
> > > of Hadoop, or is it a standalone library? Is there any publications
> > > specific
> > > on scheduling in Hadoop? Could you please share some details about
> > > scheduling or suggest some literature of Hadoop?
> > >
> > > Thanks!
> > >
> > > Best wishes,
> > >
> > > Huijun Zhu
> > >
> >
> >
> >
> > --
> > Ahmad Humayun
> > Research Assistant
> > Computer Science Dpt., LUMS
> > +92 321 4457315
> >
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315
Re: About the support of MapReduce task scheduling schemes
Posted by Zhu Huijun <na...@gmail.com>.
Thank you, Ahmad Humayun. I am not asking for the idea of MapReduce. What I
am asking is the scheduling scheme in Hadoop. I am wondering whether the
scheduling module is a part of any library implemented in Hadoop, or the
scheduling of Hadoop is based on the infrastructure under which Hadoop is
installed.
2008/2/13, Ahmad Humayun <ah...@gmail.com>:
>
> The ideas were originally published as MapReduce by Jeffrey
> Dean<http://labs.google.com/people/jeff>and Sanjay
> Ghemawat <http://labs.google.com/people/sanjay> at Google. Have a look at:
> http://labs.google.com/papers/mapreduce.html
>
>
> regards,
>
> On Feb 13, 2008 9:21 PM, Zhu Huijun <na...@gmail.com> wrote:
>
> > Dear all,
> >
> > Recently we are invesgating dynamic scheduling of parallel IO-intensive
> > applications in large-scale cluster. We are interested in Hadoop,
> > especially
> > its task scheduling schemes. Is the scheduling module a part of any
> > library
> > of Hadoop, or is it a standalone library? Is there any publications
> > specific
> > on scheduling in Hadoop? Could you please share some details about
> > scheduling or suggest some literature of Hadoop?
> >
> > Thanks!
> >
> > Best wishes,
> >
> > Huijun Zhu
> >
>
>
>
> --
> Ahmad Humayun
> Research Assistant
> Computer Science Dpt., LUMS
> +92 321 4457315
>
Re: About the support of MapReduce task scheduling schemes
Posted by Ahmad Humayun <ah...@gmail.com>.
The ideas were originally published as MapReduce by Jeffrey
Dean<http://labs.google.com/people/jeff>and Sanjay
Ghemawat <http://labs.google.com/people/sanjay> at Google. Have a look at:
http://labs.google.com/papers/mapreduce.html
regards,
On Feb 13, 2008 9:21 PM, Zhu Huijun <na...@gmail.com> wrote:
> Dear all,
>
> Recently we are invesgating dynamic scheduling of parallel IO-intensive
> applications in large-scale cluster. We are interested in Hadoop,
> especially
> its task scheduling schemes. Is the scheduling module a part of any
> library
> of Hadoop, or is it a standalone library? Is there any publications
> specific
> on scheduling in Hadoop? Could you please share some details about
> scheduling or suggest some literature of Hadoop?
>
> Thanks!
>
> Best wishes,
>
> Huijun Zhu
>
--
Ahmad Humayun
Research Assistant
Computer Science Dpt., LUMS
+92 321 4457315