You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Tobin Baker <td...@cs.washington.edu> on 2015/09/26 02:06:25 UTC
CORFU on REEF?
Hi, I've been interested in CORFU since my days at AWS, and I saw there was
a talk on some kind of CORFU implementation on REEF:
http://research.microsoft.com/apps/pubs/default.aspx?id=227128. There's no
link to any paper, though; are there any materials I can read/watch?
Thanks,
Tobin
Re: CORFU on REEF?
Posted by Tobin Baker <td...@cs.washington.edu>.
Thanks, I don't know how I forgot that part of the SIGMOD paper! I just
went back and reviewed it.
Have you seen the CorfuDB project (https://github.com/CorfuDB/CorfuDB)?
Maybe someone will have the time to figure out how to port that to REEF...
On Sun, Sep 27, 2015 at 3:17 PM, Markus Weimer <ma...@weimo.de> wrote:
> Absolutely! We actually had plans (and a project) to work on Corfu on
> REEF back when both were still Microsoft projects. Sadly, those ideas
> didn't yet make it to the top of the priority queue. One can work on
> this from two angles:
>
> (1) Corfu as a REEF application: This is a rather straight-forward way
> to get Corfu onto YARN and Mesos clusters. Other applications then
> connect to the Corfu instance via the mechanisms established in Corfu.
> The main addition would be master discovery, but that can be handled
> via the RM's list of application masters. We've written about this as
> part of our SIGMOD paper[0].
>
> (2) Corfu as a REEF service: In this way, Corfu would run on the very
> same Evaluators the app uses, albeit as a background thread. This is
> supremely desirable when considering Driver fault tolerance: One could
> log the Driver's state onto the Evaluators. That way, the Driver's
> checkpoints share fate with the Job, which makes bookkeeping a lot
> easier. As far as I know, no one has been able to act on this idea
> thus far.
>
> Markus
>
> ---
> [0]: http://www.markusweimer.com/publication/2015/06/02/Apache-REEF/
>
Re: CORFU on REEF?
Posted by Tobin Baker <td...@cs.washington.edu>.
Interesting, I wonder if that product will actually be using raw flash per
the original paper (the GitHub project uses a RocksDB or flat file
backend)...
On Sun, Sep 27, 2015 at 3:45 PM, Byung-Gon Chun <bg...@gmail.com> wrote:
> Tobin,
>
> Markus gave good descriptions.
>
> I heard that Dahlia's building a product around CORFU at VMWare.
> It'd be interesting to find out how she's using REEF.
>
> Thanks.
> -Gon
>
>
>
> On Mon, Sep 28, 2015 at 7:17 AM, Markus Weimer <ma...@weimo.de> wrote:
>
> > Absolutely! We actually had plans (and a project) to work on Corfu on
> > REEF back when both were still Microsoft projects. Sadly, those ideas
> > didn't yet make it to the top of the priority queue. One can work on
> > this from two angles:
> >
> > (1) Corfu as a REEF application: This is a rather straight-forward way
> > to get Corfu onto YARN and Mesos clusters. Other applications then
> > connect to the Corfu instance via the mechanisms established in Corfu.
> > The main addition would be master discovery, but that can be handled
> > via the RM's list of application masters. We've written about this as
> > part of our SIGMOD paper[0].
> >
> > (2) Corfu as a REEF service: In this way, Corfu would run on the very
> > same Evaluators the app uses, albeit as a background thread. This is
> > supremely desirable when considering Driver fault tolerance: One could
> > log the Driver's state onto the Evaluators. That way, the Driver's
> > checkpoints share fate with the Job, which makes bookkeeping a lot
> > easier. As far as I know, no one has been able to act on this idea
> > thus far.
> >
> > Markus
> >
> > ---
> > [0]: http://www.markusweimer.com/publication/2015/06/02/Apache-REEF/
> >
>
>
>
> --
> Byung-Gon Chun
>
Re: CORFU on REEF?
Posted by Byung-Gon Chun <bg...@gmail.com>.
Tobin,
Markus gave good descriptions.
I heard that Dahlia's building a product around CORFU at VMWare.
It'd be interesting to find out how she's using REEF.
Thanks.
-Gon
On Mon, Sep 28, 2015 at 7:17 AM, Markus Weimer <ma...@weimo.de> wrote:
> Absolutely! We actually had plans (and a project) to work on Corfu on
> REEF back when both were still Microsoft projects. Sadly, those ideas
> didn't yet make it to the top of the priority queue. One can work on
> this from two angles:
>
> (1) Corfu as a REEF application: This is a rather straight-forward way
> to get Corfu onto YARN and Mesos clusters. Other applications then
> connect to the Corfu instance via the mechanisms established in Corfu.
> The main addition would be master discovery, but that can be handled
> via the RM's list of application masters. We've written about this as
> part of our SIGMOD paper[0].
>
> (2) Corfu as a REEF service: In this way, Corfu would run on the very
> same Evaluators the app uses, albeit as a background thread. This is
> supremely desirable when considering Driver fault tolerance: One could
> log the Driver's state onto the Evaluators. That way, the Driver's
> checkpoints share fate with the Job, which makes bookkeeping a lot
> easier. As far as I know, no one has been able to act on this idea
> thus far.
>
> Markus
>
> ---
> [0]: http://www.markusweimer.com/publication/2015/06/02/Apache-REEF/
>
--
Byung-Gon Chun
Re: CORFU on REEF?
Posted by Markus Weimer <ma...@weimo.de>.
Absolutely! We actually had plans (and a project) to work on Corfu on
REEF back when both were still Microsoft projects. Sadly, those ideas
didn't yet make it to the top of the priority queue. One can work on
this from two angles:
(1) Corfu as a REEF application: This is a rather straight-forward way
to get Corfu onto YARN and Mesos clusters. Other applications then
connect to the Corfu instance via the mechanisms established in Corfu.
The main addition would be master discovery, but that can be handled
via the RM's list of application masters. We've written about this as
part of our SIGMOD paper[0].
(2) Corfu as a REEF service: In this way, Corfu would run on the very
same Evaluators the app uses, albeit as a background thread. This is
supremely desirable when considering Driver fault tolerance: One could
log the Driver's state onto the Evaluators. That way, the Driver's
checkpoints share fate with the Job, which makes bookkeeping a lot
easier. As far as I know, no one has been able to act on this idea
thus far.
Markus
---
[0]: http://www.markusweimer.com/publication/2015/06/02/Apache-REEF/