You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by chandler song <ch...@gmail.com> on 2014/01/19 14:26:46 UTC
Question about Yarn
hi all
I have some question about yarn when I read the tutorial on the website.
1) the contain is physical or logic? for example, there are three
PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
all the time. or the container will work on different PC at different time.
2) the contain, I can think it's a virtual PC which can run java
application? is my correct?
3)about mapreduce, how mapreduce run on yarn? after reading the tutorial, I
think yarn and mapreduce is totally different thing. I think the basic unit
of yarn is container. map and reduce's basic unit is map and reduce.
or how yarn handle concurrent? I know in mapreduce, I don't need to think
too much about concurrent. because mapreduce will do this for you. it will
split data into a small unit and you can do what you do. but I don't find
yarn has same thing.
Re: Question about Yarn
Posted by sudhakara st <su...@gmail.com>.
Hello Chandler,
Yarn contains separate layers for resource management, scheduling and
map-reduce. Only scheduling and resource management layer is separated to
new daemons , changed and extended in the YARN . The map reduce
functionality and processing data(execution) in parallel framework remain
same as in MR1.
I suggest you to read more on YARN architecture
http://hortonworks.com/hadoop/yarn/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
On Mon, Jan 20, 2014 at 6:42 AM, chandler song <ch...@gmail.com>wrote:
>
> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>
> this one,
>
>
> 2014/1/19 Marco Shaw <ma...@gmail.com>
>
>> Can you clarify?
>>
>> What tutorial and specific sections are you referring to?
>>
>> Marco
>>
>> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
>> wrote:
>> >
>> > hi all
>> >
>> > I have some question about yarn when I read the tutorial on the
>> website.
>> >
>> > 1) the contain is physical or logic? for example, there are three
>> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
>> all the time. or the container will work on different PC at different time.
>> >
>> > 2) the contain, I can think it's a virtual PC which can run java
>> application? is my correct?
>> >
>> > 3)about mapreduce, how mapreduce run on yarn? after reading the
>> tutorial, I think yarn and mapreduce is totally different thing. I think
>> the basic unit of yarn is container. map and reduce's basic unit is map and
>> reduce.
>> >
>> > or how yarn handle concurrent? I know in mapreduce, I don't need to
>> think too much about concurrent. because mapreduce will do this for you. it
>> will split data into a small unit and you can do what you do. but I don't
>> find yarn has same thing.
>> >
>>
>
>
--
Regards,
...Sudhakara.st
Re: Question about Yarn
Posted by sudhakara st <su...@gmail.com>.
Hello Chandler,
Yarn contains separate layers for resource management, scheduling and
map-reduce. Only scheduling and resource management layer is separated to
new daemons , changed and extended in the YARN . The map reduce
functionality and processing data(execution) in parallel framework remain
same as in MR1.
I suggest you to read more on YARN architecture
http://hortonworks.com/hadoop/yarn/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
On Mon, Jan 20, 2014 at 6:42 AM, chandler song <ch...@gmail.com>wrote:
>
> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>
> this one,
>
>
> 2014/1/19 Marco Shaw <ma...@gmail.com>
>
>> Can you clarify?
>>
>> What tutorial and specific sections are you referring to?
>>
>> Marco
>>
>> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
>> wrote:
>> >
>> > hi all
>> >
>> > I have some question about yarn when I read the tutorial on the
>> website.
>> >
>> > 1) the contain is physical or logic? for example, there are three
>> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
>> all the time. or the container will work on different PC at different time.
>> >
>> > 2) the contain, I can think it's a virtual PC which can run java
>> application? is my correct?
>> >
>> > 3)about mapreduce, how mapreduce run on yarn? after reading the
>> tutorial, I think yarn and mapreduce is totally different thing. I think
>> the basic unit of yarn is container. map and reduce's basic unit is map and
>> reduce.
>> >
>> > or how yarn handle concurrent? I know in mapreduce, I don't need to
>> think too much about concurrent. because mapreduce will do this for you. it
>> will split data into a small unit and you can do what you do. but I don't
>> find yarn has same thing.
>> >
>>
>
>
--
Regards,
...Sudhakara.st
Re: Question about Yarn
Posted by sudhakara st <su...@gmail.com>.
Hello Chandler,
Yarn contains separate layers for resource management, scheduling and
map-reduce. Only scheduling and resource management layer is separated to
new daemons , changed and extended in the YARN . The map reduce
functionality and processing data(execution) in parallel framework remain
same as in MR1.
I suggest you to read more on YARN architecture
http://hortonworks.com/hadoop/yarn/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
On Mon, Jan 20, 2014 at 6:42 AM, chandler song <ch...@gmail.com>wrote:
>
> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>
> this one,
>
>
> 2014/1/19 Marco Shaw <ma...@gmail.com>
>
>> Can you clarify?
>>
>> What tutorial and specific sections are you referring to?
>>
>> Marco
>>
>> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
>> wrote:
>> >
>> > hi all
>> >
>> > I have some question about yarn when I read the tutorial on the
>> website.
>> >
>> > 1) the contain is physical or logic? for example, there are three
>> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
>> all the time. or the container will work on different PC at different time.
>> >
>> > 2) the contain, I can think it's a virtual PC which can run java
>> application? is my correct?
>> >
>> > 3)about mapreduce, how mapreduce run on yarn? after reading the
>> tutorial, I think yarn and mapreduce is totally different thing. I think
>> the basic unit of yarn is container. map and reduce's basic unit is map and
>> reduce.
>> >
>> > or how yarn handle concurrent? I know in mapreduce, I don't need to
>> think too much about concurrent. because mapreduce will do this for you. it
>> will split data into a small unit and you can do what you do. but I don't
>> find yarn has same thing.
>> >
>>
>
>
--
Regards,
...Sudhakara.st
Re: Question about Yarn
Posted by sudhakara st <su...@gmail.com>.
Hello Chandler,
Yarn contains separate layers for resource management, scheduling and
map-reduce. Only scheduling and resource management layer is separated to
new daemons , changed and extended in the YARN . The map reduce
functionality and processing data(execution) in parallel framework remain
same as in MR1.
I suggest you to read more on YARN architecture
http://hortonworks.com/hadoop/yarn/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
On Mon, Jan 20, 2014 at 6:42 AM, chandler song <ch...@gmail.com>wrote:
>
> http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
>
> this one,
>
>
> 2014/1/19 Marco Shaw <ma...@gmail.com>
>
>> Can you clarify?
>>
>> What tutorial and specific sections are you referring to?
>>
>> Marco
>>
>> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
>> wrote:
>> >
>> > hi all
>> >
>> > I have some question about yarn when I read the tutorial on the
>> website.
>> >
>> > 1) the contain is physical or logic? for example, there are three
>> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
>> all the time. or the container will work on different PC at different time.
>> >
>> > 2) the contain, I can think it's a virtual PC which can run java
>> application? is my correct?
>> >
>> > 3)about mapreduce, how mapreduce run on yarn? after reading the
>> tutorial, I think yarn and mapreduce is totally different thing. I think
>> the basic unit of yarn is container. map and reduce's basic unit is map and
>> reduce.
>> >
>> > or how yarn handle concurrent? I know in mapreduce, I don't need to
>> think too much about concurrent. because mapreduce will do this for you. it
>> will split data into a small unit and you can do what you do. but I don't
>> find yarn has same thing.
>> >
>>
>
>
--
Regards,
...Sudhakara.st
Re: Question about Yarn
Posted by chandler song <ch...@gmail.com>.
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
this one,
2014/1/19 Marco Shaw <ma...@gmail.com>
> Can you clarify?
>
> What tutorial and specific sections are you referring to?
>
> Marco
>
> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
> wrote:
> >
> > hi all
> >
> > I have some question about yarn when I read the tutorial on the
> website.
> >
> > 1) the contain is physical or logic? for example, there are three
> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
> all the time. or the container will work on different PC at different time.
> >
> > 2) the contain, I can think it's a virtual PC which can run java
> application? is my correct?
> >
> > 3)about mapreduce, how mapreduce run on yarn? after reading the
> tutorial, I think yarn and mapreduce is totally different thing. I think
> the basic unit of yarn is container. map and reduce's basic unit is map and
> reduce.
> >
> > or how yarn handle concurrent? I know in mapreduce, I don't need to
> think too much about concurrent. because mapreduce will do this for you. it
> will split data into a small unit and you can do what you do. but I don't
> find yarn has same thing.
> >
>
Re: Question about Yarn
Posted by chandler song <ch...@gmail.com>.
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
this one,
2014/1/19 Marco Shaw <ma...@gmail.com>
> Can you clarify?
>
> What tutorial and specific sections are you referring to?
>
> Marco
>
> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
> wrote:
> >
> > hi all
> >
> > I have some question about yarn when I read the tutorial on the
> website.
> >
> > 1) the contain is physical or logic? for example, there are three
> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
> all the time. or the container will work on different PC at different time.
> >
> > 2) the contain, I can think it's a virtual PC which can run java
> application? is my correct?
> >
> > 3)about mapreduce, how mapreduce run on yarn? after reading the
> tutorial, I think yarn and mapreduce is totally different thing. I think
> the basic unit of yarn is container. map and reduce's basic unit is map and
> reduce.
> >
> > or how yarn handle concurrent? I know in mapreduce, I don't need to
> think too much about concurrent. because mapreduce will do this for you. it
> will split data into a small unit and you can do what you do. but I don't
> find yarn has same thing.
> >
>
Re: Question about Yarn
Posted by chandler song <ch...@gmail.com>.
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
this one,
2014/1/19 Marco Shaw <ma...@gmail.com>
> Can you clarify?
>
> What tutorial and specific sections are you referring to?
>
> Marco
>
> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
> wrote:
> >
> > hi all
> >
> > I have some question about yarn when I read the tutorial on the
> website.
> >
> > 1) the contain is physical or logic? for example, there are three
> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
> all the time. or the container will work on different PC at different time.
> >
> > 2) the contain, I can think it's a virtual PC which can run java
> application? is my correct?
> >
> > 3)about mapreduce, how mapreduce run on yarn? after reading the
> tutorial, I think yarn and mapreduce is totally different thing. I think
> the basic unit of yarn is container. map and reduce's basic unit is map and
> reduce.
> >
> > or how yarn handle concurrent? I know in mapreduce, I don't need to
> think too much about concurrent. because mapreduce will do this for you. it
> will split data into a small unit and you can do what you do. but I don't
> find yarn has same thing.
> >
>
Re: Question about Yarn
Posted by chandler song <ch...@gmail.com>.
http://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html
this one,
2014/1/19 Marco Shaw <ma...@gmail.com>
> Can you clarify?
>
> What tutorial and specific sections are you referring to?
>
> Marco
>
> > On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com>
> wrote:
> >
> > hi all
> >
> > I have some question about yarn when I read the tutorial on the
> website.
> >
> > 1) the contain is physical or logic? for example, there are three
> PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC
> all the time. or the container will work on different PC at different time.
> >
> > 2) the contain, I can think it's a virtual PC which can run java
> application? is my correct?
> >
> > 3)about mapreduce, how mapreduce run on yarn? after reading the
> tutorial, I think yarn and mapreduce is totally different thing. I think
> the basic unit of yarn is container. map and reduce's basic unit is map and
> reduce.
> >
> > or how yarn handle concurrent? I know in mapreduce, I don't need to
> think too much about concurrent. because mapreduce will do this for you. it
> will split data into a small unit and you can do what you do. but I don't
> find yarn has same thing.
> >
>
Re: Question about Yarn
Posted by Marco Shaw <ma...@gmail.com>.
Can you clarify?
What tutorial and specific sections are you referring to?
Marco
> On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com> wrote:
>
> hi all
>
> I have some question about yarn when I read the tutorial on the website.
>
> 1) the contain is physical or logic? for example, there are three PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC all the time. or the container will work on different PC at different time.
>
> 2) the contain, I can think it's a virtual PC which can run java application? is my correct?
>
> 3)about mapreduce, how mapreduce run on yarn? after reading the tutorial, I think yarn and mapreduce is totally different thing. I think the basic unit of yarn is container. map and reduce's basic unit is map and reduce.
>
> or how yarn handle concurrent? I know in mapreduce, I don't need to think too much about concurrent. because mapreduce will do this for you. it will split data into a small unit and you can do what you do. but I don't find yarn has same thing.
>
Re: Question about Yarn
Posted by Marco Shaw <ma...@gmail.com>.
Can you clarify?
What tutorial and specific sections are you referring to?
Marco
> On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com> wrote:
>
> hi all
>
> I have some question about yarn when I read the tutorial on the website.
>
> 1) the contain is physical or logic? for example, there are three PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC all the time. or the container will work on different PC at different time.
>
> 2) the contain, I can think it's a virtual PC which can run java application? is my correct?
>
> 3)about mapreduce, how mapreduce run on yarn? after reading the tutorial, I think yarn and mapreduce is totally different thing. I think the basic unit of yarn is container. map and reduce's basic unit is map and reduce.
>
> or how yarn handle concurrent? I know in mapreduce, I don't need to think too much about concurrent. because mapreduce will do this for you. it will split data into a small unit and you can do what you do. but I don't find yarn has same thing.
>
Re: Question about Yarn
Posted by Marco Shaw <ma...@gmail.com>.
Can you clarify?
What tutorial and specific sections are you referring to?
Marco
> On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com> wrote:
>
> hi all
>
> I have some question about yarn when I read the tutorial on the website.
>
> 1) the contain is physical or logic? for example, there are three PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC all the time. or the container will work on different PC at different time.
>
> 2) the contain, I can think it's a virtual PC which can run java application? is my correct?
>
> 3)about mapreduce, how mapreduce run on yarn? after reading the tutorial, I think yarn and mapreduce is totally different thing. I think the basic unit of yarn is container. map and reduce's basic unit is map and reduce.
>
> or how yarn handle concurrent? I know in mapreduce, I don't need to think too much about concurrent. because mapreduce will do this for you. it will split data into a small unit and you can do what you do. but I don't find yarn has same thing.
>
Re: Question about Yarn
Posted by Marco Shaw <ma...@gmail.com>.
Can you clarify?
What tutorial and specific sections are you referring to?
Marco
> On Jan 19, 2014, at 9:26 AM, chandler song <ch...@gmail.com> wrote:
>
> hi all
>
> I have some question about yarn when I read the tutorial on the website.
>
> 1) the contain is physical or logic? for example, there are three PCs(A,B,C) on Cluster. if I allocate one container. it will run on one PC all the time. or the container will work on different PC at different time.
>
> 2) the contain, I can think it's a virtual PC which can run java application? is my correct?
>
> 3)about mapreduce, how mapreduce run on yarn? after reading the tutorial, I think yarn and mapreduce is totally different thing. I think the basic unit of yarn is container. map and reduce's basic unit is map and reduce.
>
> or how yarn handle concurrent? I know in mapreduce, I don't need to think too much about concurrent. because mapreduce will do this for you. it will split data into a small unit and you can do what you do. but I don't find yarn has same thing.
>