You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Byung-Gon Chun <bg...@gmail.com> on 2015/06/01 14:06:18 UTC
creating a reef subproject or an experimental branch
Hi all,
John, Brian, and I are working with Zhengping (MSR) and Junwen (Fudan
University) on building a layer that exploits high-churn spare capacity of
data centers. We'll build it on REEF, so we would like to build it in
Apache REEF, which is an open-source environment, from the beginning.
What do you think of creating a reef subproject or an experimental reef
branch for this project?
I'd appreciate your feedback.
Thanks!
-Gon
--
Byung-Gon Chun
Re: creating a reef subproject or an experimental branch
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
> Unless we hear objection, shall we go with this approach?
+1. The general idea seems to have been accepted by lazy consent :-)
We can defer further discussions until we have a JIRA and pull request.
Speaking of which: I suggest that we let the first pull request sit out
there for 48h before merging to give community members in all time zones
a chance to look at it and comment.
Markus
Re: creating a reef subproject or an experimental branch
Posted by Byung-Gon Chun <bg...@gmail.com>.
John, this sounds good.
Unless we hear objection, shall we go with this approach?
On Thu, Jun 4, 2015 at 5:10 PM, John Yang <jo...@gmail.com> wrote:
> Hi Markus,
>
> 'reef-applications/vortex' and a new 'Component' you suggested sound good.
> Yes, any forms of help would definitely be greatly appreciated, including
> code review/merge.
>
> Thanks,
> John
>
>
> On Thu, Jun 4, 2015 at 5:02 PM, Zhengping Qian <zh...@microsoft.com>
> wrote:
>
> > Thanks Gon for looping me in. Thanks Markus/John for your
> > support/suggestions.
> >
> > // Resend the following message in case you didn't get it (sorry for any
> > duplication) as I've just realized that I might have to subscribe to the
> > mailing list (which I just did) in order to send to the list.
> >
> >
> > I’d like to share with you a bit more background/motivation for the
> > project, just FYI.
> >
> > * Today’s datacenters are often under-utilized, especially those that
> host
> > performance-sensitive tenants for on-line services, where fluctuating
> loads
> > make things even worse. On the other hand, (big) data analytical jobs
> like
> > Hadoop/Spark need increasingly more computing resources.
> >
> > * A straightforward approach is to co-host data analytical jobs with
> those
> > primary tenants. But in order not to violate performance SLAs, data
> > analytical jobs, which are run as best effort, often have to be preempted
> > and/or throttled quite frequently (even with good performance isolation).
> > Simply running Hadoop/Spark jobs is far from optimal as there could be
> > significant amount of waste due to such preemption/throttling.
> >
> > * Our hope is to have something in REEF that eases the
> > programming/execution for data analytical jobs that could better utilize
> > those high-churn spare resources (= to maximize utilization while
> > minimizing waste).
> >
> > * We will probably use some concrete computation/application to drive our
> > design and evaluation, and so I really like the idea (as John proposed)
> of
> > starting with a REEF application (= for prototyping).
> >
> > * We will follow all the rules (which I’m still learning :-) as you’ve
> > suggested.
> >
> > Thanks!
> >
> > Best,
> > Zhengping
> >
> >
> >
> >
> >
> >
> > On 6/4/15, 3:52 PM, "Markus Weimer" <ma...@weimo.de> wrote:
> >
> > >Hi,
> > >
> > >> We aim to build an application on top of REEF, so I guess the
> > >> subproject method is a better fit for our project.
> > >
> > >Agreed.
> > >
> > >> It would be great if we could use a subproject directory called
> > >> vortex, which is our project name, to host our code.
> > >
> > >How about `reef-applications/vortex`? That way we have a place for
> > >future projects of a similar kind. Will the code be in Java or C#?
> > >
> > >> Then we can file related JIRA issues under the umbrella JIRA you
> > >> suggested and create corresponding PRs for pushing code to the
> > >> directory.
> > >
> > >Yes, we can also have a new "Component" in JIRA for this, just like we
> > >have for "Driver", "REEF.NET" and so on.
> > >
> > >> Mostly Professor Chun, Brian and I can do the merging, but other
> > >> community members are also welcome to do so as well.
> > >
> > >I'd aim for something stronger than that. Given that the code will be
> > >maintained by us, we should strive for broad understanding of all code
> > >parts. Otherwise, we set ourselves up with impossible situations when we
> > >want to do breaking changes in REEF. Without understanding the apps, we
> > >might have to delete them at that time. One way to do that is to have
> > >the new code be accompanied by new committers, of course :-)
> > >
> > >> I do expect we’ll also touch on some of REEF’s core components along
> > >> the way. In such cases, I guess we could also simply follow the
> > >> standard JIRA issue and PR procedure and get feedback/help from
> > >> other members of the REEF community.
> > >
> > >Yes, this is like any other change we do today.
> > >
> > >Markus
> >
>
--
Byung-Gon Chun
Re: creating a reef subproject or an experimental branch
Posted by John Yang <jo...@gmail.com>.
Hi Markus,
'reef-applications/vortex' and a new 'Component' you suggested sound good.
Yes, any forms of help would definitely be greatly appreciated, including
code review/merge.
Thanks,
John
On Thu, Jun 4, 2015 at 5:02 PM, Zhengping Qian <zh...@microsoft.com>
wrote:
> Thanks Gon for looping me in. Thanks Markus/John for your
> support/suggestions.
>
> // Resend the following message in case you didn't get it (sorry for any
> duplication) as I've just realized that I might have to subscribe to the
> mailing list (which I just did) in order to send to the list.
>
>
> I’d like to share with you a bit more background/motivation for the
> project, just FYI.
>
> * Today’s datacenters are often under-utilized, especially those that host
> performance-sensitive tenants for on-line services, where fluctuating loads
> make things even worse. On the other hand, (big) data analytical jobs like
> Hadoop/Spark need increasingly more computing resources.
>
> * A straightforward approach is to co-host data analytical jobs with those
> primary tenants. But in order not to violate performance SLAs, data
> analytical jobs, which are run as best effort, often have to be preempted
> and/or throttled quite frequently (even with good performance isolation).
> Simply running Hadoop/Spark jobs is far from optimal as there could be
> significant amount of waste due to such preemption/throttling.
>
> * Our hope is to have something in REEF that eases the
> programming/execution for data analytical jobs that could better utilize
> those high-churn spare resources (= to maximize utilization while
> minimizing waste).
>
> * We will probably use some concrete computation/application to drive our
> design and evaluation, and so I really like the idea (as John proposed) of
> starting with a REEF application (= for prototyping).
>
> * We will follow all the rules (which I’m still learning :-) as you’ve
> suggested.
>
> Thanks!
>
> Best,
> Zhengping
>
>
>
>
>
>
> On 6/4/15, 3:52 PM, "Markus Weimer" <ma...@weimo.de> wrote:
>
> >Hi,
> >
> >> We aim to build an application on top of REEF, so I guess the
> >> subproject method is a better fit for our project.
> >
> >Agreed.
> >
> >> It would be great if we could use a subproject directory called
> >> vortex, which is our project name, to host our code.
> >
> >How about `reef-applications/vortex`? That way we have a place for
> >future projects of a similar kind. Will the code be in Java or C#?
> >
> >> Then we can file related JIRA issues under the umbrella JIRA you
> >> suggested and create corresponding PRs for pushing code to the
> >> directory.
> >
> >Yes, we can also have a new "Component" in JIRA for this, just like we
> >have for "Driver", "REEF.NET" and so on.
> >
> >> Mostly Professor Chun, Brian and I can do the merging, but other
> >> community members are also welcome to do so as well.
> >
> >I'd aim for something stronger than that. Given that the code will be
> >maintained by us, we should strive for broad understanding of all code
> >parts. Otherwise, we set ourselves up with impossible situations when we
> >want to do breaking changes in REEF. Without understanding the apps, we
> >might have to delete them at that time. One way to do that is to have
> >the new code be accompanied by new committers, of course :-)
> >
> >> I do expect we’ll also touch on some of REEF’s core components along
> >> the way. In such cases, I guess we could also simply follow the
> >> standard JIRA issue and PR procedure and get feedback/help from
> >> other members of the REEF community.
> >
> >Yes, this is like any other change we do today.
> >
> >Markus
>
Re: creating a reef subproject or an experimental branch
Posted by Zhengping Qian <zh...@microsoft.com>.
Resend to correct the link for Common Architectures using AWS Spot Instances; sorry about that.
On 6/8/15, 11:25 AM, "Zhengping Qian" <zh...@microsoft.com> wrote:
>Thanks much Markus. Please see my rough thoughts embedded below.
>
>Best, ZP
>
>
>On 6/8/15, 8:43 AM, "Markus Weimer" <ma...@weimo.de> wrote:
>
>
>
>>On 2015-06-04 19:06, Zhengping Qian wrote:
>>> In my understanding, reservations help the scheduler better
>>> prioritize resources among different and/or future jobs, while in our
>>> case, we’re trying to better leverage any spotty, preemptable
>>> resources available for low-priority/best-effort jobs.
>>
>>Agreed. But reservations can add a dimension of knowledge to the
>>best-effort jobs: Not only do you have containers that *may* be
>>preempted, but also some that *will* be.
>
>ZP> Agreed. That knowledge will be very helpful I think. I was actually wondering if it could be useful to decouple the design into two parts/steps. In step 1 (see below) we try to come up with a design without such knowledge; and in step 2, we could explore ways to refine/improve the design by integrating with YARN (and having such knowledge).
>
>>
>>> Ideally, it’d be great if we could handle those preemptions just
>>> like failures, without any cooperation between the scheduler and the
>>> job. [...] I guess we’re trying to understand/explore a
>>> non-cooperative mode for this.
>>
>>Ah, so you wouldn't want to rely on the RM giving you a grace period,
>>but instead be really good about having tasks killed at high frequency?
>>
>>Decreasing the task size seems like the go-to approach to that. I
>>believe the initial MapReduce paper pointed that out as well. Assuming
>>one can drive down the startup overhead of tasks (REEF helps with that),
>>I can see this work nicely for stateless tasks like Mappers.
>>
>>Stateful tasks like Reducers do present a challenge, though, as their
>>startup overhead is not just an engineering challenge; the state also
>>needs to be re-[loaded|computed]. That's why the preemption work in
>>Hadoop MapReduce focussed on checkpointing the Reducers. Incidentally,
>>that very same code also is in `reef-checkpointing`.
>
>ZP> Agreed. I think what you’ve pointed out is probably true when we only have resources with high failure rate. But in our scenario (sorry for I wasn’t very clear), I guess, we might in addition have a few reliable resources as well, say, by adding a few servers in our clusters.
>
>In this case, we might be able to handle stateful computation more easily. For example, we could have a stateful Reducer run on some reliable resources while it further decomposes itself into many small tasks leveraging transient, idle resources in the cluster.
>
>Scheduling of the small tasks might be another interesting angle (where we could do things differently): I guess we could leverage duplicate execution much more aggressively. Here’s a bit reasoning/intuition for that:
>
>There're essentially there ways to handle failures/preemption:
> - rollback
> - checkpointing
> - replication
>
>If you look at how people are using spotty instances in AWS today (http://aws.amazon.com/ec2/purchasing-options/spot-instances/spot-and-science/ <http://blog.celingest.com/en/2014/07/22/aws-spot-instances-uncovered/>):
> - MapReduce Architecture: rollback at task granularity
> - "Grid" Architecture: rollback at job granularity
> - Queue-Based Architecture: MapReduce-like; for tasks without dependencies
> - Checkpointing-Based Architecture
>
>No one seems using replication.
>
>However, I feel replication (or duplicate execution) could be very effective in our case, as it might lead to fewer resource waste (compared to rollback/checkpointing; if we cancel extra duplicates after task completion), given a high failure rate, though at the cost of increasing resource use (within a same time range); but for spotty resources, they might be cheap enough. :)
>
>Does that make sense?
>
>
>>
>>Markus
Re: creating a reef subproject or an experimental branch
Posted by Byung-Gon Chun <bg...@gmail.com>.
Markus,
Yes. I was thinking a maven submodule.
On Mon, Jun 8, 2015 at 12:48 PM, Markus Weimer <ma...@weimo.de> wrote:
> Thanks, Chris!
>
> On 2015-06-08 13:38, Mattmann, Chris A (3980) wrote:
> > There are no sub “projects” at the ASF per-se, there can be
> sub-“products”,
> > which are released, have a version, are released by 3 +1 votes of the
> > PMC and more +1s than -1s. They can be released from branches, and stuff
> > like that, so that part is fine.
>
> I think we used the term `sub project` here in the sense of `maven
> submodule`.
>
> Can the others please comment?
>
> Thanks,
>
> Markus
>
--
Byung-Gon Chun
Re: creating a reef subproject or an experimental branch
Posted by John Yang <jo...@gmail.com>.
Sure, if others are okay with it.
Thanks,
John
On Mon, Jun 8, 2015 at 1:17 PM, Markus Weimer <ma...@weimo.de> wrote:
>
>
> On 2015-06-08 14:11, John Yang wrote:
> > Chris, thanks for the information!
> > Markus, yes 'maven submodule' was also what I was referring to. I was
> also
> > thinking of using the master branch.
>
> Hence, this would be released as part of our release cycle, right?
>
> Markus
>
>
Re: creating a reef subproject or an experimental branch
Posted by Byung-Gon Chun <bg...@gmail.com>.
I'm fine with that.
The core part is a new fine-grained runtime layer that'll be built on the
current REEF abstraction.
It makes sense to release as part of the REEF release cycle.
On Mon, Jun 8, 2015 at 1:17 PM, Markus Weimer <ma...@weimo.de> wrote:
>
>
> On 2015-06-08 14:11, John Yang wrote:
> > Chris, thanks for the information!
> > Markus, yes 'maven submodule' was also what I was referring to. I was
> also
> > thinking of using the master branch.
>
> Hence, this would be released as part of our release cycle, right?
>
> Markus
>
>
--
Byung-Gon Chun
Re: creating a reef subproject or an experimental branch
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-06-08 14:11, John Yang wrote:
> Chris, thanks for the information!
> Markus, yes 'maven submodule' was also what I was referring to. I was also
> thinking of using the master branch.
Hence, this would be released as part of our release cycle, right?
Markus
Re: creating a reef subproject or an experimental branch
Posted by John Yang <jo...@gmail.com>.
Hi,
I filed an umbrella JIRA issue for the project:
https://issues.apache.org/jira/browse/REEF-364
I also filed a PR for the initial directory and maven submodule setup:
https://github.com/apache/incubator-reef/pull/214
Please take a look and let us know what you think.
Thanks,
John
On Mon, Jun 8, 2015 at 2:45 PM, Mattmann, Chris A (3980) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> +1, that makes sense to me :-) Thanks for the clarification.
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Associate Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> -----Original Message-----
> From: John Yang <jo...@gmail.com>
> Reply-To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
> Date: Sunday, June 7, 2015 at 9:11 PM
> To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
> Subject: Re: creating a reef subproject or an experimental branch
>
> >Chris, thanks for the information!
> >Markus, yes 'maven submodule' was also what I was referring to. I was also
> >thinking of using the master branch.
> >
> >Thanks,
> >John
> >
> >
> >On Mon, Jun 8, 2015 at 12:48 PM, Markus Weimer <ma...@weimo.de> wrote:
> >
> >> Thanks, Chris!
> >>
> >> On 2015-06-08 13:38, Mattmann, Chris A (3980) wrote:
> >> > There are no sub “projects” at the ASF per-se, there can be
> >> sub-“products”,
> >> > which are released, have a version, are released by 3 +1 votes of the
> >> > PMC and more +1s than -1s. They can be released from branches, and
> >>stuff
> >> > like that, so that part is fine.
> >>
> >> I think we used the term `sub project` here in the sense of `maven
> >> submodule`.
> >>
> >> Can the others please comment?
> >>
> >> Thanks,
> >>
> >> Markus
> >>
>
>
Re: creating a reef subproject or an experimental branch
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
+1, that makes sense to me :-) Thanks for the clarification.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: John Yang <jo...@gmail.com>
Reply-To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
Date: Sunday, June 7, 2015 at 9:11 PM
To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
Subject: Re: creating a reef subproject or an experimental branch
>Chris, thanks for the information!
>Markus, yes 'maven submodule' was also what I was referring to. I was also
>thinking of using the master branch.
>
>Thanks,
>John
>
>
>On Mon, Jun 8, 2015 at 12:48 PM, Markus Weimer <ma...@weimo.de> wrote:
>
>> Thanks, Chris!
>>
>> On 2015-06-08 13:38, Mattmann, Chris A (3980) wrote:
>> > There are no sub “projects” at the ASF per-se, there can be
>> sub-“products”,
>> > which are released, have a version, are released by 3 +1 votes of the
>> > PMC and more +1s than -1s. They can be released from branches, and
>>stuff
>> > like that, so that part is fine.
>>
>> I think we used the term `sub project` here in the sense of `maven
>> submodule`.
>>
>> Can the others please comment?
>>
>> Thanks,
>>
>> Markus
>>
Re: creating a reef subproject or an experimental branch
Posted by John Yang <jo...@gmail.com>.
Chris, thanks for the information!
Markus, yes 'maven submodule' was also what I was referring to. I was also
thinking of using the master branch.
Thanks,
John
On Mon, Jun 8, 2015 at 12:48 PM, Markus Weimer <ma...@weimo.de> wrote:
> Thanks, Chris!
>
> On 2015-06-08 13:38, Mattmann, Chris A (3980) wrote:
> > There are no sub “projects” at the ASF per-se, there can be
> sub-“products”,
> > which are released, have a version, are released by 3 +1 votes of the
> > PMC and more +1s than -1s. They can be released from branches, and stuff
> > like that, so that part is fine.
>
> I think we used the term `sub project` here in the sense of `maven
> submodule`.
>
> Can the others please comment?
>
> Thanks,
>
> Markus
>
Re: creating a reef subproject or an experimental branch
Posted by Markus Weimer <ma...@weimo.de>.
Thanks, Chris!
On 2015-06-08 13:38, Mattmann, Chris A (3980) wrote:
> There are no sub “projects” at the ASF per-se, there can be sub-“products”,
> which are released, have a version, are released by 3 +1 votes of the
> PMC and more +1s than -1s. They can be released from branches, and stuff
> like that, so that part is fine.
I think we used the term `sub project` here in the sense of `maven
submodule`.
Can the others please comment?
Thanks,
Markus
Re: creating a reef subproject or an experimental branch
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Hi Folks,
There are no sub “projects” at the ASF per-se, there can be sub-“products”,
which are released, have a version, are released by 3 +1 votes of the
PMC and more +1s than -1s. They can be released from branches, and stuff
like that, so that part is fine.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Zhengping Qian <zh...@microsoft.com>
Reply-To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
Date: Sunday, June 7, 2015 at 8:25 PM
To: "dev@reef.incubator.apache.org" <de...@reef.incubator.apache.org>
Subject: Re: creating a reef subproject or an experimental branch
>Thanks much Markus. Please see my rough thoughts embedded below.
>
>Best, ZP
>
>
>On 6/8/15, 8:43 AM, "Markus Weimer" <ma...@weimo.de> wrote:
>
>
>
>>On 2015-06-04 19:06, Zhengping Qian wrote:
>>> In my understanding, reservations help the scheduler better
>>> prioritize resources among different and/or future jobs, while in our
>>> case, we’re trying to better leverage any spotty, preemptable
>>> resources available for low-priority/best-effort jobs.
>>
>>Agreed. But reservations can add a dimension of knowledge to the
>>best-effort jobs: Not only do you have containers that *may* be
>>preempted, but also some that *will* be.
>
>ZP> Agreed. That knowledge will be very helpful I think. I was actually
>wondering if it could be useful to decouple the design into two
>parts/steps. In step 1 (see below) we try to come up with a design
>without such knowledge; and in step 2, we could explore ways to
>refine/improve the design by integrating with YARN (and having such
>knowledge).
>
>>
>>> Ideally, it’d be great if we could handle those preemptions just
>>> like failures, without any cooperation between the scheduler and the
>>> job. [...] I guess we’re trying to understand/explore a
>>> non-cooperative mode for this.
>>
>>Ah, so you wouldn't want to rely on the RM giving you a grace period,
>>but instead be really good about having tasks killed at high frequency?
>>
>>Decreasing the task size seems like the go-to approach to that. I
>>believe the initial MapReduce paper pointed that out as well. Assuming
>>one can drive down the startup overhead of tasks (REEF helps with that),
>>I can see this work nicely for stateless tasks like Mappers.
>>
>>Stateful tasks like Reducers do present a challenge, though, as their
>>startup overhead is not just an engineering challenge; the state also
>>needs to be re-[loaded|computed]. That's why the preemption work in
>>Hadoop MapReduce focussed on checkpointing the Reducers. Incidentally,
>>that very same code also is in `reef-checkpointing`.
>
>ZP> Agreed. I think what you’ve pointed out is probably true when we only
>have resources with high failure rate. But in our scenario (sorry for I
>wasn’t very clear), I guess, we might in addition have a few reliable
>resources as well, say, by adding a few servers in our clusters.
>
>In this case, we might be able to handle stateful computation more
>easily. For example, we could have a stateful Reducer run on some
>reliable resources while it further decomposes itself into many small
>tasks leveraging transient, idle resources in the cluster.
>
>Scheduling of the small tasks might be another interesting angle (where
>we could do things differently): I guess we could leverage duplicate
>execution much more aggressively. Here’s a bit reasoning/intuition for
>that:
>
>There're essentially there ways to handle failures/preemption:
> - rollback
> - checkpointing
> - replication
>
>If you look at how people are using spotty instances in AWS today
>(http://blog.celingest.com/en/2014/07/22/aws-spot-instances-uncovered/):
> - MapReduce Architecture: rollback at task granularity
> - "Grid" Architecture: rollback at job granularity
> - Queue-Based Architecture: MapReduce-like; for tasks without
>dependencies
> - Checkpointing-Based Architecture
>
>No one seems using replication.
>
>
>However, I feel replication (or duplicate execution) could be very
>effective in our case, as it might lead to fewer resource waste (compared
>to rollback/checkpointing; if we cancel extra duplicates after task
>completion), given a high failure rate, though at the cost of increasing
>resource use (within a same time range); but for spotty resources, they
>might be cheap enough. :)
>
>Does that make sense?
>
>
>>
>>Markus
Re: creating a reef subproject or an experimental branch
Posted by Zhengping Qian <zh...@microsoft.com>.
Thanks much Markus. Please see my rough thoughts embedded below.
Best, ZP
On 6/8/15, 8:43 AM, "Markus Weimer" <ma...@weimo.de> wrote:
>On 2015-06-04 19:06, Zhengping Qian wrote:
>> In my understanding, reservations help the scheduler better
>> prioritize resources among different and/or future jobs, while in our
>> case, we’re trying to better leverage any spotty, preemptable
>> resources available for low-priority/best-effort jobs.
>
>Agreed. But reservations can add a dimension of knowledge to the
>best-effort jobs: Not only do you have containers that *may* be
>preempted, but also some that *will* be.
ZP> Agreed. That knowledge will be very helpful I think. I was actually wondering if it could be useful to decouple the design into two parts/steps. In step 1 (see below) we try to come up with a design without such knowledge; and in step 2, we could explore ways to refine/improve the design by integrating with YARN (and having such knowledge).
>
>> Ideally, it’d be great if we could handle those preemptions just
>> like failures, without any cooperation between the scheduler and the
>> job. [...] I guess we’re trying to understand/explore a
>> non-cooperative mode for this.
>
>Ah, so you wouldn't want to rely on the RM giving you a grace period,
>but instead be really good about having tasks killed at high frequency?
>
>Decreasing the task size seems like the go-to approach to that. I
>believe the initial MapReduce paper pointed that out as well. Assuming
>one can drive down the startup overhead of tasks (REEF helps with that),
>I can see this work nicely for stateless tasks like Mappers.
>
>Stateful tasks like Reducers do present a challenge, though, as their
>startup overhead is not just an engineering challenge; the state also
>needs to be re-[loaded|computed]. That's why the preemption work in
>Hadoop MapReduce focussed on checkpointing the Reducers. Incidentally,
>that very same code also is in `reef-checkpointing`.
ZP> Agreed. I think what you’ve pointed out is probably true when we only have resources with high failure rate. But in our scenario (sorry for I wasn’t very clear), I guess, we might in addition have a few reliable resources as well, say, by adding a few servers in our clusters.
In this case, we might be able to handle stateful computation more easily. For example, we could have a stateful Reducer run on some reliable resources while it further decomposes itself into many small tasks leveraging transient, idle resources in the cluster.
Scheduling of the small tasks might be another interesting angle (where we could do things differently): I guess we could leverage duplicate execution much more aggressively. Here’s a bit reasoning/intuition for that:
There're essentially there ways to handle failures/preemption:
- rollback
- checkpointing
- replication
If you look at how people are using spotty instances in AWS today (http://blog.celingest.com/en/2014/07/22/aws-spot-instances-uncovered/):
- MapReduce Architecture: rollback at task granularity
- "Grid" Architecture: rollback at job granularity
- Queue-Based Architecture: MapReduce-like; for tasks without dependencies
- Checkpointing-Based Architecture
No one seems using replication.
However, I feel replication (or duplicate execution) could be very effective in our case, as it might lead to fewer resource waste (compared to rollback/checkpointing; if we cancel extra duplicates after task completion), given a high failure rate, though at the cost of increasing resource use (within a same time range); but for spotty resources, they might be cheap enough. :)
Does that make sense?
>
>Markus
Re: creating a reef subproject or an experimental branch
Posted by Markus Weimer <ma...@weimo.de>.
On 2015-06-04 19:06, Zhengping Qian wrote:
> In my understanding, reservations help the scheduler better
> prioritize resources among different and/or future jobs, while in our
> case, we’re trying to better leverage any spotty, preemptable
> resources available for low-priority/best-effort jobs.
Agreed. But reservations can add a dimension of knowledge to the
best-effort jobs: Not only do you have containers that *may* be
preempted, but also some that *will* be.
> Ideally, it’d be great if we could handle those preemptions just
> like failures, without any cooperation between the scheduler and the
> job. [...] I guess we’re trying to understand/explore a
> non-cooperative mode for this.
Ah, so you wouldn't want to rely on the RM giving you a grace period,
but instead be really good about having tasks killed at high frequency?
Decreasing the task size seems like the go-to approach to that. I
believe the initial MapReduce paper pointed that out as well. Assuming
one can drive down the startup overhead of tasks (REEF helps with that),
I can see this work nicely for stateless tasks like Mappers.
Stateful tasks like Reducers do present a challenge, though, as their
startup overhead is not just an engineering challenge; the state also
needs to be re-[loaded|computed]. That's why the preemption work in
Hadoop MapReduce focussed on checkpointing the Reducers. Incidentally,
that very same code also is in `reef-checkpointing`.
Markus
Re: creating a reef subproject or an experimental branch
Posted by Zhengping Qian <zh...@microsoft.com>.
Thanks a lot Markus for the great comments/questions. They’re very helpful!
Please see my comments embedded below, just FYI.
Best,
Zhengping
On 6/4/15, 4:29 PM, "Markus Weimer" <ma...@weimo.de> wrote:
>Hi,
>
>thanks for the details!
>
>> * A straightforward approach is to co-host data analytical jobs with
>> those primary tenants.
>
>Are you familiar with the recent work over on YARN to support resource
>reservations? That seems to address a related issue, as it can guarantee
>resources to those primary tenants.
ZP> I have some familiarity with that work; I think they’re quite complementary. Iin my understanding, reservations help the scheduler better prioritize resources among different and/or future jobs, while in our case, we’re trying to better leverage any spotty, preemptable resources available for low-priority/best-effort jobs.
>
>> Simply running Hadoop/Spark jobs is far from optimal as there could
>> be significant amount of waste due to such preemption/throttling.
>
>Can you provide an example for this? Assuming that those frameworks can
>do work-preserving preemption, life could be good, right? What are the
>situations where it breaks?
ZP> I guess work-preserving preemption (e.g., checkpointing-based approach) could be a good solution, though there’re some possible shortcomings:
- it could introduce a delay for the high-priority jobs to reclaim resources;
- it might depend on having some sort of reliable resources for suspense/resume;
- there could be some (extra) programming effort in order to preserve work, especially when this needs to be done at any given time point during execution, for arbitrary stateful computation.
Ideally, it’d be great if we could handle those preemptions just like failures, without any cooperation between the scheduler and the job. But MapReduce-like frameworks seem not designed for such a case where there could be failures/preemptions of a very high frequency (= we’re working on some simulation to validate this point). So, we’d like to investigate whether it is possible to somehow restructure the computation somewhat differently to achieve the same goal.
>
>> * Our hope is to have something in REEF that eases the
>> programming/execution for data analytical jobs that could better
>> utilize those high-churn spare resources.
>
>One natural docking point for this is to actually add preemption events
>to REEF. YARN has preemption, where it sends "reverse" resource requests
>to the application master. We've long planned on surfacing those to REEF
>applications, and maybe provide help to the application in choosing
>which containers / evaluators to vacate in response to such an
>preemption request.
>
>Is this something you have in mind?
ZP> I believe this is very helpful, especially when the job can be suspended/resumes with significantly low overhead. As described above, I guess we’re trying to understand/explore a non-cooperative mode for this.
>
>> * We will probably use some concrete computation/application to
>> drive our design and evaluation, and so I really like the idea (as
>> John proposed) of starting with a REEF application (= for
>> prototyping).
>
>Which kind of application do you have in mind? I could see some log
>compaction or other background job to be a good candidate for the
>secondary user of the cluster. Or, more advanced, we could think of
>machine learning training jobs that make due with whatever resources are
>available, e.g. for ever progressive parameter sweeps.
ZP> I feel it could depend on the restrictions we might have to introduce to the programming/computation model. I guess we might start with something very simple, say a job consisting of a set of independent tasks, and then we’ll gradually introduce more complex structures, e.g., tasks with dependencies, relational operators (like Hive), etc. I hope eventually we could support a core set of operators upon which one could build any data analytical jobs such as machine learning as you’ve mentioned.
>
>Markus
>
Re: creating a reef subproject or an experimental branch
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
thanks for the details!
> * A straightforward approach is to co-host data analytical jobs with
> those primary tenants.
Are you familiar with the recent work over on YARN to support resource
reservations? That seems to address a related issue, as it can guarantee
resources to those primary tenants.
> Simply running Hadoop/Spark jobs is far from optimal as there could
> be significant amount of waste due to such preemption/throttling.
Can you provide an example for this? Assuming that those frameworks can
do work-preserving preemption, life could be good, right? What are the
situations where it breaks?
> * Our hope is to have something in REEF that eases the
> programming/execution for data analytical jobs that could better
> utilize those high-churn spare resources.
One natural docking point for this is to actually add preemption events
to REEF. YARN has preemption, where it sends "reverse" resource requests
to the application master. We've long planned on surfacing those to REEF
applications, and maybe provide help to the application in choosing
which containers / evaluators to vacate in response to such an
preemption request.
Is this something you have in mind?
> * We will probably use some concrete computation/application to
> drive our design and evaluation, and so I really like the idea (as
> John proposed) of starting with a REEF application (= for
> prototyping).
Which kind of application do you have in mind? I could see some log
compaction or other background job to be a good candidate for the
secondary user of the cluster. Or, more advanced, we could think of
machine learning training jobs that make due with whatever resources are
available, e.g. for ever progressive parameter sweeps.
Markus
Re: creating a reef subproject or an experimental branch
Posted by Zhengping Qian <zh...@microsoft.com>.
Thanks Gon for looping me in. Thanks Markus/John for your support/suggestions.
// Resend the following message in case you didn't get it (sorry for any duplication) as I've just realized that I might have to subscribe to the mailing list (which I just did) in order to send to the list.
I’d like to share with you a bit more background/motivation for the project, just FYI.
* Today’s datacenters are often under-utilized, especially those that host performance-sensitive tenants for on-line services, where fluctuating loads make things even worse. On the other hand, (big) data analytical jobs like Hadoop/Spark need increasingly more computing resources.
* A straightforward approach is to co-host data analytical jobs with those primary tenants. But in order not to violate performance SLAs, data analytical jobs, which are run as best effort, often have to be preempted and/or throttled quite frequently (even with good performance isolation). Simply running Hadoop/Spark jobs is far from optimal as there could be significant amount of waste due to such preemption/throttling.
* Our hope is to have something in REEF that eases the programming/execution for data analytical jobs that could better utilize those high-churn spare resources (= to maximize utilization while minimizing waste).
* We will probably use some concrete computation/application to drive our design and evaluation, and so I really like the idea (as John proposed) of starting with a REEF application (= for prototyping).
* We will follow all the rules (which I’m still learning :-) as you’ve suggested.
Thanks!
Best,
Zhengping
On 6/4/15, 3:52 PM, "Markus Weimer" <ma...@weimo.de> wrote:
>Hi,
>
>> We aim to build an application on top of REEF, so I guess the
>> subproject method is a better fit for our project.
>
>Agreed.
>
>> It would be great if we could use a subproject directory called
>> vortex, which is our project name, to host our code.
>
>How about `reef-applications/vortex`? That way we have a place for
>future projects of a similar kind. Will the code be in Java or C#?
>
>> Then we can file related JIRA issues under the umbrella JIRA you
>> suggested and create corresponding PRs for pushing code to the
>> directory.
>
>Yes, we can also have a new "Component" in JIRA for this, just like we
>have for "Driver", "REEF.NET" and so on.
>
>> Mostly Professor Chun, Brian and I can do the merging, but other
>> community members are also welcome to do so as well.
>
>I'd aim for something stronger than that. Given that the code will be
>maintained by us, we should strive for broad understanding of all code
>parts. Otherwise, we set ourselves up with impossible situations when we
>want to do breaking changes in REEF. Without understanding the apps, we
>might have to delete them at that time. One way to do that is to have
>the new code be accompanied by new committers, of course :-)
>
>> I do expect we’ll also touch on some of REEF’s core components along
>> the way. In such cases, I guess we could also simply follow the
>> standard JIRA issue and PR procedure and get feedback/help from
>> other members of the REEF community.
>
>Yes, this is like any other change we do today.
>
>Markus
Re: creating a reef subproject or an experimental branch
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
> We aim to build an application on top of REEF, so I guess the
> subproject method is a better fit for our project.
Agreed.
> It would be great if we could use a subproject directory called
> vortex, which is our project name, to host our code.
How about `reef-applications/vortex`? That way we have a place for
future projects of a similar kind. Will the code be in Java or C#?
> Then we can file related JIRA issues under the umbrella JIRA you
> suggested and create corresponding PRs for pushing code to the
> directory.
Yes, we can also have a new "Component" in JIRA for this, just like we
have for "Driver", "REEF.NET" and so on.
> Mostly Professor Chun, Brian and I can do the merging, but other
> community members are also welcome to do so as well.
I'd aim for something stronger than that. Given that the code will be
maintained by us, we should strive for broad understanding of all code
parts. Otherwise, we set ourselves up with impossible situations when we
want to do breaking changes in REEF. Without understanding the apps, we
might have to delete them at that time. One way to do that is to have
the new code be accompanied by new committers, of course :-)
> I do expect we’ll also touch on some of REEF’s core components along
> the way. In such cases, I guess we could also simply follow the
> standard JIRA issue and PR procedure and get feedback/help from
> other members of the REEF community.
Yes, this is like any other change we do today.
Markus
Re: creating a reef subproject or an experimental branch
Posted by John Yang <jo...@gmail.com>.
Hi Markus,
First of all, thank you for your support!
We aim to build an application on top of REEF, so I guess the subproject
method is a better fit for our project. It would be great if we could use a
subproject directory called vortex, which is our project name, to host our
code. Then we can file related JIRA issues under the umbrella JIRA you
suggested and create corresponding PRs for pushing code to the directory.
Mostly Professor Chun, Brian and I can do the merging, but other community
members are also welcome to do so as well.
This prevents one giant pull request to master, as we will file a PR
whenever we have self-contained and test-covered code(e.g. our first
iteration of the application code) that addresses a JIRA issue. Also, since
the development is done in open, hopefully even those who are not
contributing directly to the project can understand better how the code
came to be.
I do expect we’ll also touch on some of REEF’s core components along the
way. In such cases, I guess we could also simply follow the standard JIRA
issue and PR procedure and get feedback/help from other members of the REEF
community.
What are your thoughts?
Thanks,
John
On Tue, Jun 2, 2015 at 8:01 AM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Thanks for the encouragement, Markus!
>
> I'm adding Zhengping and Junwen for discussion.
>
> On Tue, Jun 2, 2015 at 6:20 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> > Hi,
> >
> > I'm all for broadening our base :-)
> >
> > > John, Brian, and I are working with Zhengping (MSR) and Junwen (Fudan
> > > University) on building a layer that exploits high-churn spare capacity
> > of
> > > data centers.
> >
> > Do you have some more descriptions of what you plan on doing? Or is it
> > too early to be concrete?
> >
> >
> We'd be able to share more details once we make some progress.
>
>
>
> > > What do you think of creating a reef subproject or an experimental reef
> > > branch for this project?
> >
> > Which one is more appropriate depends a lot on what the project is about
> > in terms of REEF terms. Is this more of a REEF application or a new set
> > of features for REEF itself? If it is the former, making it a
> > sub-project, e.g. in a newly formed `reef-applications` sounds like the
> > right approach. If it is the latter, maybe it can live in a branch.
> >
> >
> This's my opinion. It depends on what the end result is.
> Making it as a subproject seems to be a good starting point. But if the
> project becomes relevant to the reef core, it'd be good to merge this
> project codebase back to the main reef branch.
>
>
>
> > Either way, we have to be careful with the social aspects of this. It
> > shouldn't be done in isolation and at some point show up as one giant
> > pull request. It would be awesome if we could structure it in a way
> > where the new work shows up as a series of small changes to REEF, all
> > part of an umbrella JIRA.
> >
> >
> Yes. I agree.
>
> Thanks.
> -Gon
>
>
>
> > Markus
> >
> >
>
>
> --
> Byung-Gon Chun
>
Re: creating a reef subproject or an experimental branch
Posted by Byung-Gon Chun <bg...@gmail.com>.
Thanks for the encouragement, Markus!
I'm adding Zhengping and Junwen for discussion.
On Tue, Jun 2, 2015 at 6:20 AM, Markus Weimer <ma...@weimo.de> wrote:
> Hi,
>
> I'm all for broadening our base :-)
>
> > John, Brian, and I are working with Zhengping (MSR) and Junwen (Fudan
> > University) on building a layer that exploits high-churn spare capacity
> of
> > data centers.
>
> Do you have some more descriptions of what you plan on doing? Or is it
> too early to be concrete?
>
>
We'd be able to share more details once we make some progress.
> > What do you think of creating a reef subproject or an experimental reef
> > branch for this project?
>
> Which one is more appropriate depends a lot on what the project is about
> in terms of REEF terms. Is this more of a REEF application or a new set
> of features for REEF itself? If it is the former, making it a
> sub-project, e.g. in a newly formed `reef-applications` sounds like the
> right approach. If it is the latter, maybe it can live in a branch.
>
>
This's my opinion. It depends on what the end result is.
Making it as a subproject seems to be a good starting point. But if the
project becomes relevant to the reef core, it'd be good to merge this
project codebase back to the main reef branch.
> Either way, we have to be careful with the social aspects of this. It
> shouldn't be done in isolation and at some point show up as one giant
> pull request. It would be awesome if we could structure it in a way
> where the new work shows up as a series of small changes to REEF, all
> part of an umbrella JIRA.
>
>
Yes. I agree.
Thanks.
-Gon
> Markus
>
>
--
Byung-Gon Chun
Re: creating a reef subproject or an experimental branch
Posted by Markus Weimer <ma...@weimo.de>.
Hi,
I'm all for broadening our base :-)
> John, Brian, and I are working with Zhengping (MSR) and Junwen (Fudan
> University) on building a layer that exploits high-churn spare capacity of
> data centers.
Do you have some more descriptions of what you plan on doing? Or is it
too early to be concrete?
> What do you think of creating a reef subproject or an experimental reef
> branch for this project?
Which one is more appropriate depends a lot on what the project is about
in terms of REEF terms. Is this more of a REEF application or a new set
of features for REEF itself? If it is the former, making it a
sub-project, e.g. in a newly formed `reef-applications` sounds like the
right approach. If it is the latter, maybe it can live in a branch.
Either way, we have to be careful with the social aspects of this. It
shouldn't be done in isolation and at some point show up as one giant
pull request. It would be awesome if we could structure it in a way
where the new work shows up as a series of small changes to REEF, all
part of an umbrella JIRA.
Markus