You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by 牛兆捷 <nz...@gmail.com> on 2013/06/21 07:29:17 UTC
About container preemption
https://issues.apache.org/jira/browse/YARN-45 support work-preserve
preemption
Has the AM support container's state checkpoint now?
Does the checkpoint support map, shuffle and reduce phase?
--
*Sincerely,*
*Zhaojie*
*
*
Re: About container preemption
Posted by 牛兆捷 <nz...@gmail.com>.
Thanks~
For reduce phase, if the reduce has finished all the input and began to run
a algorithm, how to implement the checkpoint for it?
2013/6/22 Chris Douglas <cd...@apache.org>
> The APIs for checkpointing are in branch-2, but support for the
> capacity scheduler and the MapReduce implementation are making their
> way through review.
>
> As of now, the checkpointing is for the shuffle and reduce phases,
> only. Map checkpointing requires cooperation with the RecordReader
> (i.e., the symmetric operation to create an InputSplit from a
> RecordReader) which has not been implemented. -C
>
> On Thu, Jun 20, 2013 at 10:29 PM, 牛兆捷 <nz...@gmail.com> wrote:
> > https://issues.apache.org/jira/browse/YARN-45 support work-preserve
> > preemption
> >
> > Has the AM support container's state checkpoint now?
> > Does the checkpoint support map, shuffle and reduce phase?
> >
> > --
> > *Sincerely,*
> > *Zhaojie*
> > *
> > *
>
--
*Sincerely,*
*Zhaojie*
*
*
Re: About container preemption
Posted by 牛兆捷 <nz...@gmail.com>.
Thanks ~
2013/6/24 Harsh J <ha...@cloudera.com>
> Hi,
>
> You may track https://issues.apache.org/jira/browse/MAPREDUCE-5269 for
> progress on Reduce end checkpointing.
>
> On Sat, Jun 22, 2013 at 12:12 PM, 牛兆捷 <nz...@gmail.com> wrote:
> > Hi Chris:
> > I don't find the related checkpoint implementation for shuffle and reduce
> > phase of MapReduce.
> > Are the code open now?
> >
> >
> > 2013/6/22 Chris Douglas <cd...@apache.org>
> >
> >> The APIs for checkpointing are in branch-2, but support for the
> >> capacity scheduler and the MapReduce implementation are making their
> >> way through review.
> >>
> >> As of now, the checkpointing is for the shuffle and reduce phases,
> >> only. Map checkpointing requires cooperation with the RecordReader
> >> (i.e., the symmetric operation to create an InputSplit from a
> >> RecordReader) which has not been implemented. -C
> >>
> >> On Thu, Jun 20, 2013 at 10:29 PM, 牛兆捷 <nz...@gmail.com> wrote:
> >> > https://issues.apache.org/jira/browse/YARN-45 support work-preserve
> >> > preemption
> >> >
> >> > Has the AM support container's state checkpoint now?
> >> > Does the checkpoint support map, shuffle and reduce phase?
> >> >
> >> > --
> >> > *Sincerely,*
> >> > *Zhaojie*
> >> > *
> >> > *
> >>
> >
> >
> >
> > --
> > *Sincerely,*
> > *Zhaojie*
> > *
> > *
>
>
>
> --
> Harsh J
>
--
*Sincerely,*
*Zhaojie*
*
*
Re: About container preemption
Posted by Harsh J <ha...@cloudera.com>.
Hi,
You may track https://issues.apache.org/jira/browse/MAPREDUCE-5269 for
progress on Reduce end checkpointing.
On Sat, Jun 22, 2013 at 12:12 PM, 牛兆捷 <nz...@gmail.com> wrote:
> Hi Chris:
> I don't find the related checkpoint implementation for shuffle and reduce
> phase of MapReduce.
> Are the code open now?
>
>
> 2013/6/22 Chris Douglas <cd...@apache.org>
>
>> The APIs for checkpointing are in branch-2, but support for the
>> capacity scheduler and the MapReduce implementation are making their
>> way through review.
>>
>> As of now, the checkpointing is for the shuffle and reduce phases,
>> only. Map checkpointing requires cooperation with the RecordReader
>> (i.e., the symmetric operation to create an InputSplit from a
>> RecordReader) which has not been implemented. -C
>>
>> On Thu, Jun 20, 2013 at 10:29 PM, 牛兆捷 <nz...@gmail.com> wrote:
>> > https://issues.apache.org/jira/browse/YARN-45 support work-preserve
>> > preemption
>> >
>> > Has the AM support container's state checkpoint now?
>> > Does the checkpoint support map, shuffle and reduce phase?
>> >
>> > --
>> > *Sincerely,*
>> > *Zhaojie*
>> > *
>> > *
>>
>
>
>
> --
> *Sincerely,*
> *Zhaojie*
> *
> *
--
Harsh J
Re: About container preemption
Posted by 牛兆捷 <nz...@gmail.com>.
Hi Chris:
I don't find the related checkpoint implementation for shuffle and reduce
phase of MapReduce.
Are the code open now?
2013/6/22 Chris Douglas <cd...@apache.org>
> The APIs for checkpointing are in branch-2, but support for the
> capacity scheduler and the MapReduce implementation are making their
> way through review.
>
> As of now, the checkpointing is for the shuffle and reduce phases,
> only. Map checkpointing requires cooperation with the RecordReader
> (i.e., the symmetric operation to create an InputSplit from a
> RecordReader) which has not been implemented. -C
>
> On Thu, Jun 20, 2013 at 10:29 PM, 牛兆捷 <nz...@gmail.com> wrote:
> > https://issues.apache.org/jira/browse/YARN-45 support work-preserve
> > preemption
> >
> > Has the AM support container's state checkpoint now?
> > Does the checkpoint support map, shuffle and reduce phase?
> >
> > --
> > *Sincerely,*
> > *Zhaojie*
> > *
> > *
>
--
*Sincerely,*
*Zhaojie*
*
*
Re: About container preemption
Posted by Chris Douglas <cd...@apache.org>.
The APIs for checkpointing are in branch-2, but support for the
capacity scheduler and the MapReduce implementation are making their
way through review.
As of now, the checkpointing is for the shuffle and reduce phases,
only. Map checkpointing requires cooperation with the RecordReader
(i.e., the symmetric operation to create an InputSplit from a
RecordReader) which has not been implemented. -C
On Thu, Jun 20, 2013 at 10:29 PM, 牛兆捷 <nz...@gmail.com> wrote:
> https://issues.apache.org/jira/browse/YARN-45 support work-preserve
> preemption
>
> Has the AM support container's state checkpoint now?
> Does the checkpoint support map, shuffle and reduce phase?
>
> --
> *Sincerely,*
> *Zhaojie*
> *
> *